* Re: [PATCH net-next v2 1/1] net: dsa: fix fixed-link port registration
From: Marek Behun @ 2019-08-11 16:21 UTC (permalink / raw)
To: Vladimir Oltean
Cc: Heiner Kallweit, Andrew Lunn, netdev, Sebastian Reichel,
Vivien Didelot, Florian Fainelli, David S . Miller
In-Reply-To: <CA+h21hoOZQ79rj0SLZGLnkSjrKD3aLNos0GcnRjre-Ls=Tq=4w@mail.gmail.com>
On Sun, 11 Aug 2019 18:16:11 +0300
Vladimir Oltean <olteanv@gmail.com> wrote:
> The DSA fixed-link port functionality *has* been converted to phylink.
> See:
> - https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=0e27921816ad9
> - https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=7fb5a711545d7d25fe9726a9ad277474dd83bd06
>
/o\ my bad. I did not realize that I was working on 5.2 :(. Sorry.
^ permalink raw reply
* Re: [PATCH net-next 1/2] net: dsa: mv88e6xxx: fix RGMII-ID port setup
From: Marek Behun @ 2019-08-11 16:14 UTC (permalink / raw)
To: Andrew Lunn
Cc: netdev, Heiner Kallweit, Sebastian Reichel, Vivien Didelot,
Florian Fainelli, David S . Miller
In-Reply-To: <20190811153153.GB14290@lunn.ch>
On Sun, 11 Aug 2019 17:31:53 +0200
Andrew Lunn <andrew@lunn.ch> wrote:
> On Sun, Aug 11, 2019 at 05:08:11PM +0200, Marek Behún wrote:
> > The mv88e6xxx_port_setup_mac looks if one of the {link, speed, duplex}
> > parameters is being changed from the current setting, and if not, does
> > not do anything. This test is wrong in some situations: this method also
> > has the mode argument, which can also be changed.
> >
> > For example on Turris Omnia, the mode is PHY_INTERFACE_MODE_RGMII_ID,
> > which has to be set byt the ->port_set_rgmii_delay method. The test does
> > not look if mode is being changed (in fact there is currently no method
> > to determine port mode as phy_interface_t type).
> >
> > The simplest solution seems to be to drop this test altogether and
> > simply do the setup when requested.
>
> Hi Marek
>
> Unfortunately, that code is there for a reason. phylink can call the
> ->mac_config() method once per second. It is documented that
> mac_config() should only reconfigure what, if anything, has changed.
> And mv88e6xxx_port_setup_mac() needs to disable the port in order to
> change anything. So the change you propose here, under some
> conditions, will cause the port to be disabled/enables once per
> second.
>
> We need to fix this by expanding the test, not removing it. My
> current _guess_ would be, we need to add a ops->port_get_rgmii_delay()
> so we can see if that is what needs configuring.
>
> Andrew
Hi Andrew,
what if we added a phy_mode member to struct mv88e6xxx_port, and either
set it to PHY_INTERFACE_MODE_NA in mv88e6xxx_setup, or implemented
methods for converting the switch register values to
PHY_INTERFACE_MODE_* values.
This way we could just add port.mode == mode check to port_setup_mac
method.
I am willing to implement this if you think this could work.
Marek
^ permalink raw reply
* Re: [PATCH net-next 2/2] net: fixed_phy: set is_gigabit_capable member when needed
From: Marek Behun @ 2019-08-11 16:08 UTC (permalink / raw)
To: Andrew Lunn
Cc: netdev, Heiner Kallweit, Sebastian Reichel, Vivien Didelot,
Florian Fainelli, David S . Miller
In-Reply-To: <20190811154001.GC14290@lunn.ch>
On Sun, 11 Aug 2019 17:40:01 +0200
Andrew Lunn <andrew@lunn.ch> wrote:
> On Sun, Aug 11, 2019 at 05:08:12PM +0200, Marek Behún wrote:
> > The fixed_phy driver does not set the phydev->is_gigabit_capable member
> > when the fixed_phy is gigabit capable.
>
> Neither does any other PHY driver. It should be possible to tell if a
> PHY supports 1G by looking at register values. If this does not work
> for fixed_link, it means we are missing something in the emulation.
> That is what we should be fixing.
>
> Also, this change has nothing to do the lp_advertise, what you
> previously said the problem was. At the moment, i don't get the
> feeling you have really dug all the way down and really understand the
> root causes.
>
> Andrew
Andrew,
is_gigabit_capable is otherwise set only in the phy_probe function.
This function is not called at all for the DSA cpu port fixed_link phy.
Why is that? But I guess it is not important anymore, if CPU and DSA
were converted to phylink in net-next. I shall test it and let you know.
In any case, sorry for the spam.
Marek
^ permalink raw reply
* Re: [BUG] fec mdio times out under system stress
From: Andrew Lunn @ 2019-08-11 16:03 UTC (permalink / raw)
To: Vladimir Oltean
Cc: Russell King - ARM Linux admin, Florian Fainelli, netdev,
Hubert Feurstein, Fabio Estevam, linux-arm-kernel,
Heiner Kallweit
In-Reply-To: <CA+h21hqkVoQWRweKKCFdvLLOLyP4gEtXQvJ9CO_J7i+YQW+TWw@mail.gmail.com>
> I think a better question is why is the FEC MDIO controller configured
> to emit interrupts anyway (especially since the API built on top does
> not benefit in any way from this)? Hubert (copied) sent an interesting
> email very recently where he pointed out that this is one of the main
> sources of jitter when reading the PTP clock on a Marvell switch
> attached over MDIO.
Hi Vladimir
One reason is runtime power management.
For a write operation, you could set it going and return
immediately. Many drivers do, and then when the next read/write
operation comes along, they poll in a loop waiting for the device to
go idle, if it was still busy from the last operation.
However, FEC supports runtime PM. When an MDIO read/write call is
made, it calls runtime PM functions to indicate the device is active.
Once it has completed the MDIO transaction, it calls runtime PM
functions to indicate the device is inactive. These transitions can
cause clocks to be enabled/disabled. If we don't wait around for the
operation to complete, the clock could be disabled too early, and bad
things would happen.
You could replace the interrupt with a sleeping poll, but my guess
would be, that has more jitter than using an interrupt.
Andrew
^ permalink raw reply
* Re: [PATCH net-next 2/2] net: fixed_phy: set is_gigabit_capable member when needed
From: Andrew Lunn @ 2019-08-11 15:40 UTC (permalink / raw)
To: Marek Behún
Cc: netdev, Heiner Kallweit, Sebastian Reichel, Vivien Didelot,
Florian Fainelli, David S . Miller
In-Reply-To: <20190811150812.6780-2-marek.behun@nic.cz>
On Sun, Aug 11, 2019 at 05:08:12PM +0200, Marek Behún wrote:
> The fixed_phy driver does not set the phydev->is_gigabit_capable member
> when the fixed_phy is gigabit capable.
Neither does any other PHY driver. It should be possible to tell if a
PHY supports 1G by looking at register values. If this does not work
for fixed_link, it means we are missing something in the emulation.
That is what we should be fixing.
Also, this change has nothing to do the lp_advertise, what you
previously said the problem was. At the moment, i don't get the
feeling you have really dug all the way down and really understand the
root causes.
Andrew
^ permalink raw reply
* Re: [PATCH net-next 1/2] net: dsa: mv88e6xxx: fix RGMII-ID port setup
From: Andrew Lunn @ 2019-08-11 15:31 UTC (permalink / raw)
To: Marek Behún
Cc: netdev, Heiner Kallweit, Sebastian Reichel, Vivien Didelot,
Florian Fainelli, David S . Miller
In-Reply-To: <20190811150812.6780-1-marek.behun@nic.cz>
On Sun, Aug 11, 2019 at 05:08:11PM +0200, Marek Behún wrote:
> The mv88e6xxx_port_setup_mac looks if one of the {link, speed, duplex}
> parameters is being changed from the current setting, and if not, does
> not do anything. This test is wrong in some situations: this method also
> has the mode argument, which can also be changed.
>
> For example on Turris Omnia, the mode is PHY_INTERFACE_MODE_RGMII_ID,
> which has to be set byt the ->port_set_rgmii_delay method. The test does
> not look if mode is being changed (in fact there is currently no method
> to determine port mode as phy_interface_t type).
>
> The simplest solution seems to be to drop this test altogether and
> simply do the setup when requested.
Hi Marek
Unfortunately, that code is there for a reason. phylink can call the
->mac_config() method once per second. It is documented that
mac_config() should only reconfigure what, if anything, has changed.
And mv88e6xxx_port_setup_mac() needs to disable the port in order to
change anything. So the change you propose here, under some
conditions, will cause the port to be disabled/enables once per
second.
We need to fix this by expanding the test, not removing it. My
current _guess_ would be, we need to add a ops->port_get_rgmii_delay()
so we can see if that is what needs configuring.
Andrew
^ permalink raw reply
* Re: [PATCH net-next v2 1/1] net: dsa: fix fixed-link port registration
From: Andrew Lunn @ 2019-08-11 15:18 UTC (permalink / raw)
To: Marek Behun
Cc: Heiner Kallweit, netdev, Sebastian Reichel, Vivien Didelot,
Florian Fainelli, David S . Miller
In-Reply-To: <20190811160404.06450685@nic.cz>
On Sun, Aug 11, 2019 at 04:04:04PM +0200, Marek Behun wrote:
> OK guys, something is terribly wrong here.
>
> I bisected to the commit mentioned (88d6272acaaa), looked around at the
> genphy functions, tried adding the link=0 workaround and it did work,
> so I though this was the issue.
>
> What I realized now is that before the commit 88d6272acaaa things
> worked because of two bugs, which negated each other. This commit caused
> one of this bugs not to fire, and thus the second bug was not negated.
>
> What actually happened before the commit that broke it is this:
> - after the fixed_phy is created, the parameters are corrent
> - genphy_read_status breaks the parameters:
> - first it sets the parameters to unknown (SPEED_UNKNOWN,
> DUPLEX_UNKNOWN)
> - then read the registers, which are simulated for fixed_phy
> - then it uses phy-core.c:phy_resolve_aneg_linkmode function, which
> looks for correct settings by bit-anding the ->advertising and
> ->lp_advertigins bit arrays. But in fixed_phy, ->lp_advertising
> is set to zero, so the parameters are left at SPEED_UNKNOWN, ...
> (this is the first bug)
> - then adjust_link is called, which then goes to
> mv88e6xxx_port_setup_mac, where there is a test if it should change
> something:
> if (state.link == link && state.speed == speed &&
> state.duplex == duplex)
> return 0;
> - since current speed on the switch port (state.speed) is SPEED_1000,
> and new speed is SPEED_UNKNOWN, this test fails, and so the rest of
> this function is called, which makes the port work
> (the if test is the second bug)
>
> After the commit that broke things:
> - after the fixed_phy is created, the parameters are corrent
> - genphy_read_status doesn't change them
> - mv88e6xxx_port_setup_mac does nothing, since the if condition above
> is true
>
> So, there are two things that are broken:
> - the test in mv88e6xxx_port_setup_mac whether there is to be a change
> should be more sophisticated
> - fixed_phy should also simulate the lp_advertising register
>
> What do you think of this?
Marek
This is the sort of information i like. I was having trouble
understanding what was really wrong and how it was fixed by your
previous patch.
So setting the emulated lp_advertise to advertise makes a lot of sense
for fixed phy. And it is something worthy of stable.
As for mv88e6xxx_port_setup_mac(), which parameter is actually
important here? My assumption was, if one of the other parameters
changes, it would not happen alone. The link would also go down, and
later come up again, etc. But it seems that assumption is wrong.
At a guess, it is the RGMII delays. That would explain CRC errors in
frames received by the master interface.
Andrew
^ permalink raw reply
* Re: [PATCH net-next v2 1/1] net: dsa: fix fixed-link port registration
From: Vladimir Oltean @ 2019-08-11 15:16 UTC (permalink / raw)
To: Marek Behun
Cc: Heiner Kallweit, Andrew Lunn, netdev, Sebastian Reichel,
Vivien Didelot, Florian Fainelli, David S . Miller
In-Reply-To: <20190811160404.06450685@nic.cz>
Hi Marek,
On Sun, 11 Aug 2019 at 17:06, Marek Behun <marek.behun@nic.cz> wrote:
>
> OK guys, something is terribly wrong here.
>
> I bisected to the commit mentioned (88d6272acaaa), looked around at the
> genphy functions, tried adding the link=0 workaround and it did work,
> so I though this was the issue.
>
> What I realized now is that before the commit 88d6272acaaa things
> worked because of two bugs, which negated each other. This commit caused
> one of this bugs not to fire, and thus the second bug was not negated.
>
> What actually happened before the commit that broke it is this:
> - after the fixed_phy is created, the parameters are corrent
> - genphy_read_status breaks the parameters:
> - first it sets the parameters to unknown (SPEED_UNKNOWN,
> DUPLEX_UNKNOWN)
> - then read the registers, which are simulated for fixed_phy
> - then it uses phy-core.c:phy_resolve_aneg_linkmode function, which
> looks for correct settings by bit-anding the ->advertising and
> ->lp_advertigins bit arrays. But in fixed_phy, ->lp_advertising
> is set to zero, so the parameters are left at SPEED_UNKNOWN, ...
> (this is the first bug)
> - then adjust_link is called, which then goes to
> mv88e6xxx_port_setup_mac, where there is a test if it should change
> something:
> if (state.link == link && state.speed == speed &&
> state.duplex == duplex)
> return 0;
> - since current speed on the switch port (state.speed) is SPEED_1000,
> and new speed is SPEED_UNKNOWN, this test fails, and so the rest of
> this function is called, which makes the port work
> (the if test is the second bug)
>
> After the commit that broke things:
> - after the fixed_phy is created, the parameters are corrent
> - genphy_read_status doesn't change them
> - mv88e6xxx_port_setup_mac does nothing, since the if condition above
> is true
>
> So, there are two things that are broken:
> - the test in mv88e6xxx_port_setup_mac whether there is to be a change
> should be more sophisticated
> - fixed_phy should also simulate the lp_advertising register
>
> What do you think of this?
>
I don't know. But I think Heiner was asking you what kernel you're on
because of what you said here:
> Hopefully DSA fixed-link port functionality will be converted to phylink
> API soon.
The DSA fixed-link port functionality *has* been converted to phylink.
See:
- https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=0e27921816ad9
- https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=7fb5a711545d7d25fe9726a9ad277474dd83bd06
> Marek
>
> On Sun, 11 Aug 2019 13:35:20 +0200
> Heiner Kallweit <hkallweit1@gmail.com> wrote:
>
> > On 11.08.2019 05:39, Andrew Lunn wrote:
> > > On Sun, Aug 11, 2019 at 05:18:57AM +0200, Marek Behún wrote:
> > >> Commit 88d6272acaaa ("net: phy: avoid unneeded MDIO reads in
> > >> genphy_read_status") broke fixed link DSA port registration in
> > >> dsa_port_fixed_link_register_of: the genphy_read_status does not do what
> > >> it is supposed to and the following adjust_link is given wrong
> > >> parameters.
> > >
> > > Hi Marek
> > >
> > > Which parameters are incorrect?
> > >
> > > In fixed_phy.c, __fixed_phy_register() there is:
> > >
> > > /* propagate the fixed link values to struct phy_device */
> > > phy->link = status->link;
> > > if (status->link) {
> > > phy->speed = status->speed;
> > > phy->duplex = status->duplex;
> > > phy->pause = status->pause;
> > > phy->asym_pause = status->asym_pause;
> > > }
> > >
> > > Are we not initialising something? Or is the initialisation done here
> > > getting reset sometime afterwards?
> > >
> > In addition to Andrew's question:
> > We talk about this DT config: armada-385-turris-omnia.dts ?
> > Which kernel version are you using?
> >
> > > Thanks
> > > Andrew
> > >
> > Heiner
>
Regards,
-Vladimir
^ permalink raw reply
* [PATCH 4/7] PCI/net: Use PCI_STD_NUM_BARS in loops instead of PCI_STD_RESOURCE_END
From: Denis Efremov @ 2019-08-11 15:08 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Denis Efremov, Giuseppe Cavallaro, Alexandre Torgue, netdev,
linux-pci, linux-kernel
In-Reply-To: <20190811150802.2418-1-efremov@linux.com>
This patch refactors the loop condition scheme from
'i <= PCI_STD_RESOURCE_END' to 'i < PCI_STD_NUM_BARS'.
Signed-off-by: Denis Efremov <efremov@linux.com>
---
drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c | 4 ++--
drivers/net/ethernet/synopsys/dwc-xlgmac-pci.c | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
index 86f9c07a38cf..cfe496cdd78b 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
@@ -258,7 +258,7 @@ static int stmmac_pci_probe(struct pci_dev *pdev,
}
/* Get the base address of device */
- for (i = 0; i <= PCI_STD_RESOURCE_END; i++) {
+ for (i = 0; i < PCI_STD_NUM_BARS; i++) {
if (pci_resource_len(pdev, i) == 0)
continue;
ret = pcim_iomap_regions(pdev, BIT(i), pci_name(pdev));
@@ -296,7 +296,7 @@ static void stmmac_pci_remove(struct pci_dev *pdev)
stmmac_dvr_remove(&pdev->dev);
- for (i = 0; i <= PCI_STD_RESOURCE_END; i++) {
+ for (i = 0; i < PCI_STD_NUM_BARS; i++) {
if (pci_resource_len(pdev, i) == 0)
continue;
pcim_iounmap_regions(pdev, BIT(i));
diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac-pci.c b/drivers/net/ethernet/synopsys/dwc-xlgmac-pci.c
index 386bafe74c3f..fa8604d7b797 100644
--- a/drivers/net/ethernet/synopsys/dwc-xlgmac-pci.c
+++ b/drivers/net/ethernet/synopsys/dwc-xlgmac-pci.c
@@ -34,7 +34,7 @@ static int xlgmac_probe(struct pci_dev *pcidev, const struct pci_device_id *id)
return ret;
}
- for (i = 0; i <= PCI_STD_RESOURCE_END; i++) {
+ for (i = 0; i < PCI_STD_NUM_BARS; i++) {
if (pci_resource_len(pcidev, i) == 0)
continue;
ret = pcim_iomap_regions(pcidev, BIT(i), XLGMAC_DRV_NAME);
--
2.21.0
^ permalink raw reply related
* Re: [PATCH net-next v2 1/1] net: dsa: fix fixed-link port registration
From: Marek Behun @ 2019-08-11 15:08 UTC (permalink / raw)
To: Heiner Kallweit
Cc: Andrew Lunn, netdev, Sebastian Reichel, Vivien Didelot,
Florian Fainelli, David S . Miller
In-Reply-To: <20190811160404.06450685@nic.cz>
I have sent two new patches, each fixing one of the bugs that negated
each other.
Marek
^ permalink raw reply
* [PATCH 1/7] PCI: Add define for the number of standard PCI BARs
From: Denis Efremov @ 2019-08-11 15:07 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Denis Efremov, Sebastian Ott, Gerald Schaefer, H. Peter Anvin,
Giuseppe Cavallaro, Alexandre Torgue, Matt Porter,
Alexandre Bounine, Peter Jones, Bartlomiej Zolnierkiewicz,
Cornelia Huck, Alex Williamson, kvm, linux-fbdev, netdev, x86,
linux-s390, linux-pci, linux-kernel
In-Reply-To: <20190811150802.2418-1-efremov@linux.com>
Code that iterates over all standard PCI BARs typically uses
PCI_STD_RESOURCE_END. However, it requires the "unusual" loop condition
"i <= PCI_STD_RESOURCE_END" rather than something more standard like
"i < PCI_STD_NUM_BARS".
This patch adds the definition PCI_STD_NUM_BARS which is equivalent to
"PCI_STD_RESOURCE_END + 1" and updates loop conditions to use it.
Signed-off-by: Denis Efremov <efremov@linux.com>
---
drivers/pci/quirks.c | 2 +-
include/linux/pci.h | 2 +-
include/uapi/linux/pci_regs.h | 1 +
3 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 208aacf39329..02bdf3a0231e 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -475,7 +475,7 @@ static void quirk_extend_bar_to_page(struct pci_dev *dev)
{
int i;
- for (i = 0; i <= PCI_STD_RESOURCE_END; i++) {
+ for (i = 0; i < PCI_STD_NUM_BARS; i++) {
struct resource *r = &dev->resource[i];
if (r->flags & IORESOURCE_MEM && resource_size(r) < PAGE_SIZE) {
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 9e700d9f9f28..7b9590d5dc2d 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -76,7 +76,7 @@ enum pci_mmap_state {
enum {
/* #0-5: standard PCI resources */
PCI_STD_RESOURCES,
- PCI_STD_RESOURCE_END = 5,
+ PCI_STD_RESOURCE_END = PCI_STD_RESOURCES + PCI_STD_NUM_BARS - 1,
/* #6: expansion ROM resource */
PCI_ROM_RESOURCE,
diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
index f28e562d7ca8..68b571d491eb 100644
--- a/include/uapi/linux/pci_regs.h
+++ b/include/uapi/linux/pci_regs.h
@@ -34,6 +34,7 @@
* of which the first 64 bytes are standardized as follows:
*/
#define PCI_STD_HEADER_SIZEOF 64
+#define PCI_STD_NUM_BARS 6 /* Number of standard BARs */
#define PCI_VENDOR_ID 0x00 /* 16 bits */
#define PCI_DEVICE_ID 0x02 /* 16 bits */
#define PCI_COMMAND 0x04 /* 16 bits */
--
2.21.0
^ permalink raw reply related
* [PATCH 0/7] Add definition for the number of standard PCI BARs
From: Denis Efremov @ 2019-08-11 15:07 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Denis Efremov, Sebastian Ott, Gerald Schaefer, H. Peter Anvin,
Giuseppe Cavallaro, Alexandre Torgue, Matt Porter,
Alexandre Bounine, Peter Jones, Bartlomiej Zolnierkiewicz,
Cornelia Huck, Alex Williamson, kvm, linux-fbdev, netdev, x86,
linux-s390, linux-pci, linux-kernel
Code that iterates over all standard PCI BARs typically uses
PCI_STD_RESOURCE_END, but this is error-prone because it requires
"i <= PCI_STD_RESOURCE_END" rather than something like
"i < PCI_STD_NUM_BARS". We could add such a definition and use it the same
way PCI_SRIOV_NUM_BARS is used. There is already the definition
PCI_BAR_COUNT for s390 only. Thus, this patchset introduces it globally.
The patch is splitted into 7 parts for different drivers/subsystems for
easy readability.
Denis Efremov (7):
PCI: Add define for the number of standard PCI BARs
s390/pci: Replace PCI_BAR_COUNT with PCI_STD_NUM_BARS
x86/PCI: Use PCI_STD_NUM_BARS in loops instead of PCI_STD_RESOURCE_END
PCI/net: Use PCI_STD_NUM_BARS in loops instead of PCI_STD_RESOURCE_END
rapidio/tsi721: use PCI_STD_NUM_BARS in loops instead of
PCI_STD_RESOURCE_END
efifb: Use PCI_STD_NUM_BARS in loops instead of PCI_STD_RESOURCE_END
vfio_pci: Use PCI_STD_NUM_BARS in loops instead of
PCI_STD_RESOURCE_END
arch/s390/include/asm/pci.h | 5 +----
arch/s390/include/asm/pci_clp.h | 6 +++---
arch/s390/pci/pci.c | 16 ++++++++--------
arch/s390/pci/pci_clp.c | 6 +++---
arch/x86/pci/common.c | 2 +-
drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c | 4 ++--
drivers/net/ethernet/synopsys/dwc-xlgmac-pci.c | 2 +-
drivers/pci/quirks.c | 2 +-
drivers/rapidio/devices/tsi721.c | 2 +-
drivers/vfio/pci/vfio_pci.c | 4 ++--
drivers/vfio/pci/vfio_pci_config.c | 2 +-
drivers/vfio/pci/vfio_pci_private.h | 4 ++--
drivers/video/fbdev/efifb.c | 2 +-
include/linux/pci.h | 2 +-
include/uapi/linux/pci_regs.h | 1 +
15 files changed, 29 insertions(+), 31 deletions(-)
--
2.21.0
^ permalink raw reply
* [PATCH net-next 2/2] net: fixed_phy: set is_gigabit_capable member when needed
From: Marek Behún @ 2019-08-11 15:08 UTC (permalink / raw)
To: netdev
Cc: Marek Behún, Heiner Kallweit, Sebastian Reichel,
Vivien Didelot, Andrew Lunn, Florian Fainelli, David S . Miller
In-Reply-To: <20190811150812.6780-1-marek.behun@nic.cz>
The fixed_phy driver does not set the phydev->is_gigabit_capable member
when the fixed_phy is gigabit capable. This in turn causes
phy_device.c:genphy_read_status function to return unknown phy
parameters (SPEED_UNKNOWN, DUPLEX_UNKNOWN, ...), if the fixed_phy is
created with SPEED_1000.
Signed-off-by: Marek Behún <marek.behun@nic.cz>
Cc: Heiner Kallweit <hkallweit1@gmail.com>
Cc: Sebastian Reichel <sebastian.reichel@collabora.co.uk>
Cc: Vivien Didelot <vivien.didelot@gmail.com>
Cc: Andrew Lunn <andrew@lunn.ch>
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: David S. Miller <davem@davemloft.net>
---
drivers/net/phy/fixed_phy.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/net/phy/fixed_phy.c b/drivers/net/phy/fixed_phy.c
index 3ffe46df249e..424b02ad7b7b 100644
--- a/drivers/net/phy/fixed_phy.c
+++ b/drivers/net/phy/fixed_phy.c
@@ -286,6 +286,7 @@ static struct phy_device *__fixed_phy_register(unsigned int irq,
phy->supported);
linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseT_Full_BIT,
phy->supported);
+ phy->is_gigabit_capable = 1;
/* fall through */
case SPEED_100:
linkmode_set_bit(ETHTOOL_LINK_MODE_100baseT_Half_BIT,
--
2.21.0
^ permalink raw reply related
* [PATCH net-next 1/2] net: dsa: mv88e6xxx: fix RGMII-ID port setup
From: Marek Behún @ 2019-08-11 15:08 UTC (permalink / raw)
To: netdev
Cc: Marek Behún, Heiner Kallweit, Sebastian Reichel,
Vivien Didelot, Andrew Lunn, Florian Fainelli, David S . Miller
The mv88e6xxx_port_setup_mac looks if one of the {link, speed, duplex}
parameters is being changed from the current setting, and if not, does
not do anything. This test is wrong in some situations: this method also
has the mode argument, which can also be changed.
For example on Turris Omnia, the mode is PHY_INTERFACE_MODE_RGMII_ID,
which has to be set byt the ->port_set_rgmii_delay method. The test does
not look if mode is being changed (in fact there is currently no method
to determine port mode as phy_interface_t type).
The simplest solution seems to be to drop this test altogether and
simply do the setup when requested.
Signed-off-by: Marek Behún <marek.behun@nic.cz>
Cc: Heiner Kallweit <hkallweit1@gmail.com>
Cc: Sebastian Reichel <sebastian.reichel@collabora.co.uk>
Cc: Vivien Didelot <vivien.didelot@gmail.com>
Cc: Andrew Lunn <andrew@lunn.ch>
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: David S. Miller <davem@davemloft.net>
---
drivers/net/dsa/mv88e6xxx/chip.c | 9 ---------
1 file changed, 9 deletions(-)
diff --git a/drivers/net/dsa/mv88e6xxx/chip.c b/drivers/net/dsa/mv88e6xxx/chip.c
index 2e8b1ab2c6f7..aae63f6515b3 100644
--- a/drivers/net/dsa/mv88e6xxx/chip.c
+++ b/drivers/net/dsa/mv88e6xxx/chip.c
@@ -420,15 +420,6 @@ int mv88e6xxx_port_setup_mac(struct mv88e6xxx_chip *chip, int port, int link,
if (err)
return err;
- /* Has anything actually changed? We don't expect the
- * interface mode to change without one of the other
- * parameters also changing
- */
- if (state.link == link &&
- state.speed == speed &&
- state.duplex == duplex)
- return 0;
-
/* Port's MAC control must not be changed unless the link is down */
err = chip->info->ops->port_set_link(chip, port, 0);
if (err)
--
2.21.0
^ permalink raw reply related
* Re: [BUG] fec mdio times out under system stress
From: Vladimir Oltean @ 2019-08-11 14:54 UTC (permalink / raw)
To: Russell King - ARM Linux admin
Cc: linux-arm-kernel, Fabio Estevam, netdev, Andrew Lunn,
Florian Fainelli, Heiner Kallweit, Hubert Feurstein
In-Reply-To: <20190811133707.GC13294@shell.armlinux.org.uk>
Hi Russell, Fabio,
On Sun, 11 Aug 2019 at 16:42, Russell King - ARM Linux admin
<linux@armlinux.org.uk> wrote:
>
> Hi Fabio,
>
> When I woke up this morning, I found that one of the Hummingboards
> had gone offline (as in, lost network link) during the night.
> Investigating, I find that the system had gone into OOM, and at
> that time, triggered an unrelated:
>
> [4111697.698776] fec 2188000.ethernet eth0: MDIO read timeout
> [4111697.712996] MII_DATA: 0x6006796d
> [4111697.729415] MII_SPEED: 0x0000001a
> [4111697.745232] IEVENT: 0x00000000
> [4111697.745242] IMASK: 0x0a8000aa
> [4111698.002233] Atheros 8035 ethernet 2188000.ethernet-1:00: PHY state change RUNNING -> HALTED
> [4111698.009882] fec 2188000.ethernet eth0: Link is Down
>
> This is on a dual-core iMX6.
>
> It looks like the read actually completed (since MII_DATA contains
> the register data) but we somehow lost the interrupt (or maybe
> received the interrupt after wait_for_completion_timeout() timed
> out.)
>
> From what I can see, the OOM events happened on CPU1, CPU1 was
> allocated the FEC interrupt, and the PHY polling that suffered the
> MDIO timeout was on CPU0.
>
> Given that IEVENT is zero, it seems that CPU1 had read serviced the
> interrupt, but it is not clear how far through processing that it
> was - it may be that fec_enet_interrupt() had been delayed by the
> OOM condition.
>
> This seems rather fragile - as the system slowing down due to OOM
> triggers the network to completely collapse by phylib taking the
> PHY offline, making the system inaccessible except through the
> console.
>
> In my case, even serial console wasn't operational (except for
> magic sysrq). Not sure what agetty was playing at... so the only
> way I could recover any information from the system was to connect
> the HDMI and plug in a USB keyboard.
>
> Any thoughts on how FEC MDIO accesses could be made more robust?
>
I think a better question is why is the FEC MDIO controller configured
to emit interrupts anyway (especially since the API built on top does
not benefit in any way from this)? Hubert (copied) sent an interesting
email very recently where he pointed out that this is one of the main
sources of jitter when reading the PTP clock on a Marvell switch
attached over MDIO.
> Maybe phylib should retry a number of times - but with read-sensitive
> registers, if the read has already completed successfully, and its
> just a problem with the FEC MDIO hardware, that could cause issues.
>
> Thanks.
>
> --
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
> According to speedtest.net: 11.9Mbps down 500kbps up
Regards,
-Vladimir
^ permalink raw reply
* [PATCH net v3] net: dsa: Check existence of .port_mdb_add callback before calling it
From: Chen-Yu Tsai @ 2019-08-11 14:18 UTC (permalink / raw)
To: Andrew Lunn, Vivien Didelot, Florian Fainelli, David S. Miller
Cc: Chen-Yu Tsai, netdev, linux-kernel
From: Chen-Yu Tsai <wens@csie.org>
The dsa framework has optional .port_mdb_{prepare,add,del} callback fields
for drivers to handle multicast database entries. When adding an entry, the
framework goes through a prepare phase, then a commit phase. Drivers not
providing these callbacks should be detected in the prepare phase.
DSA core may still bypass the bridge layer and call the dsa_port_mdb_add
function directly with no prepare phase or no switchdev trans object,
and the framework ends up calling an undefined .port_mdb_add callback.
This results in a NULL pointer dereference, as shown in the log below.
The other functions seem to be properly guarded. Do the same for
.port_mdb_add in dsa_switch_mdb_add_bitmap() as well.
8<--- cut here ---
Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = (ptrval)
[00000000] *pgd=00000000
Internal error: Oops: 80000005 [#1] SMP ARM
Modules linked in: rtl8xxxu rtl8192cu rtl_usb rtl8192c_common rtlwifi mac80211 cfg80211
CPU: 1 PID: 134 Comm: kworker/1:2 Not tainted 5.3.0-rc1-00247-gd3519030752a #1
Hardware name: Allwinner sun7i (A20) Family
Workqueue: events switchdev_deferred_process_work
PC is at 0x0
LR is at dsa_switch_event+0x570/0x620
pc : [<00000000>] lr : [<c08533ec>] psr: 80070013
sp : ee871db8 ip : 00000000 fp : ee98d0a4
r10: 0000000c r9 : 00000008 r8 : ee89f710
r7 : ee98d040 r6 : ee98d088 r5 : c0f04c48 r4 : ee98d04c
r3 : 00000000 r2 : ee89f710 r1 : 00000008 r0 : ee98d040
Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
Control: 10c5387d Table: 6deb406a DAC: 00000051
Process kworker/1:2 (pid: 134, stack limit = 0x(ptrval))
Stack: (0xee871db8 to 0xee872000)
1da0: ee871e14 103ace2d
1dc0: 00000000 ffffffff 00000000 ee871e14 00000005 00000000 c08524a0 00000000
1de0: ffffe000 c014bdfc c0f04c48 ee871e98 c0f04c48 ee9e5000 c0851120 c014bef0
1e00: 00000000 b643aea2 ee9b4068 c08509a8 ee2bf940 ee89f710 ee871ecb 00000000
1e20: 00000008 103ace2d 00000000 c087e248 ee29c868 103ace2d 00000001 ffffffff
1e40: 00000000 ee871e98 00000006 00000000 c0fb2a50 c087e2d0 ffffffff c08523c4
1e60: ffffffff c014bdfc 00000006 c0fad2d0 ee871e98 ee89f710 00000000 c014c500
1e80: 00000000 ee89f3c0 c0f04c48 00000000 ee9e5000 c087dfb4 ee9e5000 00000000
1ea0: ee89f710 ee871ecb 00000001 103ace2d 00000000 c0f04c48 00000000 c087e0a8
1ec0: 00000000 efd9a3e0 0089f3c0 103ace2d ee89f700 ee89f710 ee9e5000 00000122
1ee0: 00000100 c087e130 ee89f700 c0fad2c8 c1003ef0 c087de4c 2e928000 c0fad2ec
1f00: c0fad2ec ee839580 ef7a62c0 ef7a9400 00000000 c087def8 c0fad2ec c01447dc
1f20: ef315640 ef7a62c0 00000008 ee839580 ee839594 ef7a62c0 00000008 c0f03d00
1f40: ef7a62d8 ef7a62c0 ffffe000 c0145b84 ffffe000 c0fb2420 c0bfaa8c 00000000
1f60: ffffe000 ee84b600 ee84b5c0 00000000 ee870000 ee839580 c0145b40 ef0e5ea4
1f80: ee84b61c c014a6f8 00000001 ee84b5c0 c014a5b0 00000000 00000000 00000000
1fa0: 00000000 00000000 00000000 c01010e8 00000000 00000000 00000000 00000000
1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[<c08533ec>] (dsa_switch_event) from [<c014bdfc>] (notifier_call_chain+0x48/0x84)
[<c014bdfc>] (notifier_call_chain) from [<c014bef0>] (raw_notifier_call_chain+0x18/0x20)
[<c014bef0>] (raw_notifier_call_chain) from [<c08509a8>] (dsa_port_mdb_add+0x48/0x74)
[<c08509a8>] (dsa_port_mdb_add) from [<c087e248>] (__switchdev_handle_port_obj_add+0x54/0xd4)
[<c087e248>] (__switchdev_handle_port_obj_add) from [<c087e2d0>] (switchdev_handle_port_obj_add+0x8/0x14)
[<c087e2d0>] (switchdev_handle_port_obj_add) from [<c08523c4>] (dsa_slave_switchdev_blocking_event+0x94/0xa4)
[<c08523c4>] (dsa_slave_switchdev_blocking_event) from [<c014bdfc>] (notifier_call_chain+0x48/0x84)
[<c014bdfc>] (notifier_call_chain) from [<c014c500>] (blocking_notifier_call_chain+0x50/0x68)
[<c014c500>] (blocking_notifier_call_chain) from [<c087dfb4>] (switchdev_port_obj_notify+0x44/0xa8)
[<c087dfb4>] (switchdev_port_obj_notify) from [<c087e0a8>] (switchdev_port_obj_add_now+0x90/0x104)
[<c087e0a8>] (switchdev_port_obj_add_now) from [<c087e130>] (switchdev_port_obj_add_deferred+0x14/0x5c)
[<c087e130>] (switchdev_port_obj_add_deferred) from [<c087de4c>] (switchdev_deferred_process+0x64/0x104)
[<c087de4c>] (switchdev_deferred_process) from [<c087def8>] (switchdev_deferred_process_work+0xc/0x14)
[<c087def8>] (switchdev_deferred_process_work) from [<c01447dc>] (process_one_work+0x218/0x50c)
[<c01447dc>] (process_one_work) from [<c0145b84>] (worker_thread+0x44/0x5bc)
[<c0145b84>] (worker_thread) from [<c014a6f8>] (kthread+0x148/0x150)
[<c014a6f8>] (kthread) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
Exception stack(0xee871fb0 to 0xee871ff8)
1fa0: 00000000 00000000 00000000 00000000
1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Code: bad PC value
---[ end trace 1292c61abd17b130 ]---
[<c08533ec>] (dsa_switch_event) from [<c014bdfc>] (notifier_call_chain+0x48/0x84)
corresponds to
$ arm-linux-gnueabihf-addr2line -C -i -e vmlinux c08533ec
linux/net/dsa/switch.c:156
linux/net/dsa/switch.c:178
linux/net/dsa/switch.c:328
Fixes: e6db98db8a95 ("net: dsa: add switch mdb bitmap functions")
Signed-off-by: Chen-Yu Tsai <wens@csie.org>
---
Changes since v2:
- Moved the check back to dsa_switch_mdb_add_bitmap, but don't change
the function return type this time
- Rewrote the commit log
Changes since v1:
- Moved the check to the beginning of dsa_switch_mdb_add()
---
net/dsa/switch.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/net/dsa/switch.c b/net/dsa/switch.c
index 4ec5b7f85d51..09d9286b27cc 100644
--- a/net/dsa/switch.c
+++ b/net/dsa/switch.c
@@ -153,6 +153,9 @@ static void dsa_switch_mdb_add_bitmap(struct dsa_switch *ds,
{
int port;
+ if (!ds->ops->port_mdb_add)
+ return;
+
for_each_set_bit(port, bitmap, ds->num_ports)
ds->ops->port_mdb_add(ds, port, mdb);
}
--
2.20.1
^ permalink raw reply related
* Re: [BUG] fec mdio times out under system stress
From: Russell King - ARM Linux admin @ 2019-08-11 14:06 UTC (permalink / raw)
To: linux-arm-kernel, Fabio Estevam, netdev, Andrew Lunn,
Florian Fainelli, Heiner Kallweit
In-Reply-To: <20190811133707.GC13294@shell.armlinux.org.uk>
On Sun, Aug 11, 2019 at 02:37:07PM +0100, Russell King - ARM Linux admin wrote:
> Hi Fabio,
>
> When I woke up this morning, I found that one of the Hummingboards
> had gone offline (as in, lost network link) during the night.
> Investigating, I find that the system had gone into OOM, and at
> that time, triggered an unrelated:
>
> [4111697.698776] fec 2188000.ethernet eth0: MDIO read timeout
> [4111697.712996] MII_DATA: 0x6006796d
> [4111697.729415] MII_SPEED: 0x0000001a
> [4111697.745232] IEVENT: 0x00000000
> [4111697.745242] IMASK: 0x0a8000aa
> [4111698.002233] Atheros 8035 ethernet 2188000.ethernet-1:00: PHY state change RUNNING -> HALTED
> [4111698.009882] fec 2188000.ethernet eth0: Link is Down
>
> This is on a dual-core iMX6.
>
> It looks like the read actually completed (since MII_DATA contains
> the register data) but we somehow lost the interrupt (or maybe
> received the interrupt after wait_for_completion_timeout() timed
> out.)
>
> From what I can see, the OOM events happened on CPU1, CPU1 was
> allocated the FEC interrupt, and the PHY polling that suffered the
> MDIO timeout was on CPU0.
>
> Given that IEVENT is zero, it seems that CPU1 had read serviced the
> interrupt, but it is not clear how far through processing that it
> was - it may be that fec_enet_interrupt() had been delayed by the
> OOM condition.
>
> This seems rather fragile - as the system slowing down due to OOM
> triggers the network to completely collapse by phylib taking the
> PHY offline, making the system inaccessible except through the
> console.
>
> In my case, even serial console wasn't operational (except for
> magic sysrq). Not sure what agetty was playing at... so the only
> way I could recover any information from the system was to connect
> the HDMI and plug in a USB keyboard.
>
> Any thoughts on how FEC MDIO accesses could be made more robust?
>
> Maybe phylib should retry a number of times - but with read-sensitive
> registers, if the read has already completed successfully, and its
> just a problem with the FEC MDIO hardware, that could cause issues.
I should also note for the phylib people:
After phylib has received an error, and has entered the HALTED state,
downing the interface produces this warning, which seems rather unfair
as the FEC doesn't know that its PHY suffered an error - it is merely
reversing the phy_start() that happened when the interface was opened.
[4144039.099786] ------------[ cut here ]------------
[4144039.109001] WARNING: CPU: 1 PID: 25842 at drivers/net/phy/phy.c:835 fec_enet_close+0x14/0x148
[4144039.124366] called from state HALTED
[4144039.132626] Modules linked in: 8021q brcmfmac brcmutil imx_thermal
snd_soc_imx_spdif snd_soc_imx_audmux nvmem_imx_ocotp cfg80211
snd_soc_sgtl5000 imx_sdma
virt_dma rc_cec snd_soc_fsl_ssi snd_soc_fsl_spdif coda imx_pcm_dma
v4l2_mem2mem
imx_vdoa etnaviv videobuf2_dma_contig gpu_sched dw_hdmi_cec
dw_hdmi_ahb_audio imx6q_cpufreq caamrng caam_jr caam error ip_tables
x_tables [last unloaded: evbug][4144039.177249] CPU: 1 PID: 25842 Comm:
ip Tainted: G W 5.1.0+ #319
[4144039.189477] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[4144039.201025] [<c00194f0>] (unwind_backtrace) from [<c0014748>] (show_stack+0x10/0x14)
[4144039.213769] [<c0014748>] (show_stack) from [<c082e11c>] (dump_stack+0x9c/0xd4)
[4144039.225959] [<c082e11c>] (dump_stack) from [<c0030f3c>] (__warn+0xf8/0x124)
[4144039.237828] [<c0030f3c>] (__warn) from [<c0031030>] (warn_slowpath_fmt+0x38/0x48)
[4144039.250275] [<c0031030>] (warn_slowpath_fmt) from [<c052cb14>] (fec_enet_close+0x14/0x148)
[4144039.263558] [<c052cb14>] (fec_enet_close) from [<c066b5bc>] (__dev_close_many+0x88/0xf0)
[4144039.276639] [<c066b5bc>] (__dev_close_many) from [<c06740cc>] (__dev_change_flags+0xa4/0x1a0)
[4144039.290130] [<c06740cc>] (__dev_change_flags) from [<c06741e8>] (dev_change_flags+0x18/0x48)
[4144039.303558] [<c06741e8>] (dev_change_flags) from [<c068db80>] (do_setlink+0x29c/0x990)
[4144039.316517] [<c068db80>] (do_setlink) from [<c068eae8>] (__rtnl_newlink+0x4a4/0x72c)
[4144039.329282] [<c068eae8>] (__rtnl_newlink) from [<c068edb0>] (rtnl_newlink+0x40/0x60)
[4144039.342023] [<c068edb0>] (rtnl_newlink) from [<c0689a10>] (rtnetlink_rcv_msg+0x244/0x470)
[4144039.355224] [<c0689a10>] (rtnetlink_rcv_msg) from [<c06d20a0>] (netlink_rcv_skb+0xe0/0xf4)
[4144039.368492] [<c06d20a0>] (netlink_rcv_skb) from [<c06d194c>] (netlink_unicast+0x170/0x1b8)
[4144039.381758] [<c06d194c>] (netlink_unicast) from [<c06d1cdc>] (netlink_sendmsg+0x2b8/0x340)
[4144039.395023] [<c06d1cdc>] (netlink_sendmsg) from [<c064999c>] (sock_sendmsg+0x14/0x24)
[4144039.407825] [<c064999c>] (sock_sendmsg) from [<c064a5a0>] (___sys_sendmsg+0x200/0x214)
[4144039.420714] [<c064a5a0>] (___sys_sendmsg) from [<c064b0c4>] (__sys_sendmsg+0x40/0x6c)
[4144039.433506] [<c064b0c4>] (__sys_sendmsg) from [<c0009000>] (ret_fast_syscall+0x0/0x28)
[4144039.446366] Exception stack(0xe1933fa8 to 0xe1933ff0)
[4144039.456329] 3fa0: bea086f0 bea086bc 00000003 bea086d0 00000000 00000000
[4144039.469464] 3fc0: bea086f0 bea086bc 00000000 00000128 0062a278 bea086d0 5d5011f3 0062a000
[4144039.482590] 3fe0: 00000128 bea08680 b6dfc4cb b6d7d6f6
[4144039.492652] irq event stamp: 0
[4144039.500650] hardirqs last enabled at (0): [<00000000>] (null)
[4144039.511628] hardirqs last disabled at (0): [<c002e55c>] copy_process.part.4+0x30c/0x19e8
[4144039.524691] softirqs last enabled at (0): [<c002e55c>] copy_process.part.4+0x30c/0x19e8
[4144039.537691] softirqs last disabled at (0): [<00000000>] (null)
[4144039.548554] ---[ end trace b5e8d4b0f30ae00b ]---
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
^ permalink raw reply
* Re: [PATCH net-next v2 1/1] net: dsa: fix fixed-link port registration
From: Marek Behun @ 2019-08-11 14:04 UTC (permalink / raw)
To: Heiner Kallweit
Cc: Andrew Lunn, netdev, Sebastian Reichel, Vivien Didelot,
Florian Fainelli, David S . Miller
In-Reply-To: <91cd70df-c856-4c7e-7ebb-c01519fb13d2@gmail.com>
OK guys, something is terribly wrong here.
I bisected to the commit mentioned (88d6272acaaa), looked around at the
genphy functions, tried adding the link=0 workaround and it did work,
so I though this was the issue.
What I realized now is that before the commit 88d6272acaaa things
worked because of two bugs, which negated each other. This commit caused
one of this bugs not to fire, and thus the second bug was not negated.
What actually happened before the commit that broke it is this:
- after the fixed_phy is created, the parameters are corrent
- genphy_read_status breaks the parameters:
- first it sets the parameters to unknown (SPEED_UNKNOWN,
DUPLEX_UNKNOWN)
- then read the registers, which are simulated for fixed_phy
- then it uses phy-core.c:phy_resolve_aneg_linkmode function, which
looks for correct settings by bit-anding the ->advertising and
->lp_advertigins bit arrays. But in fixed_phy, ->lp_advertising
is set to zero, so the parameters are left at SPEED_UNKNOWN, ...
(this is the first bug)
- then adjust_link is called, which then goes to
mv88e6xxx_port_setup_mac, where there is a test if it should change
something:
if (state.link == link && state.speed == speed &&
state.duplex == duplex)
return 0;
- since current speed on the switch port (state.speed) is SPEED_1000,
and new speed is SPEED_UNKNOWN, this test fails, and so the rest of
this function is called, which makes the port work
(the if test is the second bug)
After the commit that broke things:
- after the fixed_phy is created, the parameters are corrent
- genphy_read_status doesn't change them
- mv88e6xxx_port_setup_mac does nothing, since the if condition above
is true
So, there are two things that are broken:
- the test in mv88e6xxx_port_setup_mac whether there is to be a change
should be more sophisticated
- fixed_phy should also simulate the lp_advertising register
What do you think of this?
Marek
On Sun, 11 Aug 2019 13:35:20 +0200
Heiner Kallweit <hkallweit1@gmail.com> wrote:
> On 11.08.2019 05:39, Andrew Lunn wrote:
> > On Sun, Aug 11, 2019 at 05:18:57AM +0200, Marek Behún wrote:
> >> Commit 88d6272acaaa ("net: phy: avoid unneeded MDIO reads in
> >> genphy_read_status") broke fixed link DSA port registration in
> >> dsa_port_fixed_link_register_of: the genphy_read_status does not do what
> >> it is supposed to and the following adjust_link is given wrong
> >> parameters.
> >
> > Hi Marek
> >
> > Which parameters are incorrect?
> >
> > In fixed_phy.c, __fixed_phy_register() there is:
> >
> > /* propagate the fixed link values to struct phy_device */
> > phy->link = status->link;
> > if (status->link) {
> > phy->speed = status->speed;
> > phy->duplex = status->duplex;
> > phy->pause = status->pause;
> > phy->asym_pause = status->asym_pause;
> > }
> >
> > Are we not initialising something? Or is the initialisation done here
> > getting reset sometime afterwards?
> >
> In addition to Andrew's question:
> We talk about this DT config: armada-385-turris-omnia.dts ?
> Which kernel version are you using?
>
> > Thanks
> > Andrew
> >
> Heiner
^ permalink raw reply
* [BUG] fec mdio times out under system stress
From: Russell King - ARM Linux admin @ 2019-08-11 13:37 UTC (permalink / raw)
To: linux-arm-kernel, Fabio Estevam, netdev, Andrew Lunn,
Florian Fainelli, Heiner Kallweit
Hi Fabio,
When I woke up this morning, I found that one of the Hummingboards
had gone offline (as in, lost network link) during the night.
Investigating, I find that the system had gone into OOM, and at
that time, triggered an unrelated:
[4111697.698776] fec 2188000.ethernet eth0: MDIO read timeout
[4111697.712996] MII_DATA: 0x6006796d
[4111697.729415] MII_SPEED: 0x0000001a
[4111697.745232] IEVENT: 0x00000000
[4111697.745242] IMASK: 0x0a8000aa
[4111698.002233] Atheros 8035 ethernet 2188000.ethernet-1:00: PHY state change RUNNING -> HALTED
[4111698.009882] fec 2188000.ethernet eth0: Link is Down
This is on a dual-core iMX6.
It looks like the read actually completed (since MII_DATA contains
the register data) but we somehow lost the interrupt (or maybe
received the interrupt after wait_for_completion_timeout() timed
out.)
From what I can see, the OOM events happened on CPU1, CPU1 was
allocated the FEC interrupt, and the PHY polling that suffered the
MDIO timeout was on CPU0.
Given that IEVENT is zero, it seems that CPU1 had read serviced the
interrupt, but it is not clear how far through processing that it
was - it may be that fec_enet_interrupt() had been delayed by the
OOM condition.
This seems rather fragile - as the system slowing down due to OOM
triggers the network to completely collapse by phylib taking the
PHY offline, making the system inaccessible except through the
console.
In my case, even serial console wasn't operational (except for
magic sysrq). Not sure what agetty was playing at... so the only
way I could recover any information from the system was to connect
the HDMI and plug in a USB keyboard.
Any thoughts on how FEC MDIO accesses could be made more robust?
Maybe phylib should retry a number of times - but with read-sensitive
registers, if the read has already completed successfully, and its
just a problem with the FEC MDIO hardware, that could cause issues.
Thanks.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
^ permalink raw reply
* Re: [PATCH net-next v2 1/1] net: dsa: fix fixed-link port registration
From: Heiner Kallweit @ 2019-08-11 11:35 UTC (permalink / raw)
To: Marek Behún
Cc: Andrew Lunn, netdev, Sebastian Reichel, Vivien Didelot,
Florian Fainelli, David S . Miller
In-Reply-To: <20190811033910.GL30120@lunn.ch>
On 11.08.2019 05:39, Andrew Lunn wrote:
> On Sun, Aug 11, 2019 at 05:18:57AM +0200, Marek Behún wrote:
>> Commit 88d6272acaaa ("net: phy: avoid unneeded MDIO reads in
>> genphy_read_status") broke fixed link DSA port registration in
>> dsa_port_fixed_link_register_of: the genphy_read_status does not do what
>> it is supposed to and the following adjust_link is given wrong
>> parameters.
>
> Hi Marek
>
> Which parameters are incorrect?
>
> In fixed_phy.c, __fixed_phy_register() there is:
>
> /* propagate the fixed link values to struct phy_device */
> phy->link = status->link;
> if (status->link) {
> phy->speed = status->speed;
> phy->duplex = status->duplex;
> phy->pause = status->pause;
> phy->asym_pause = status->asym_pause;
> }
>
> Are we not initialising something? Or is the initialisation done here
> getting reset sometime afterwards?
>
In addition to Andrew's question:
We talk about this DT config: armada-385-turris-omnia.dts ?
Which kernel version are you using?
> Thanks
> Andrew
>
Heiner
^ permalink raw reply
* Re: [PATCH] Documentation/networking/af_xdp: Inhibit reference to struct socket
From: Jonathan Neuschäfer @ 2019-08-11 11:32 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Jonathan Neuschäfer, linux-doc, David S. Miller,
Alexei Starovoitov, Daniel Borkmann, Jakub Kicinski,
Jesper Dangaard Brouer, John Fastabend, netdev, xdp-newbies, bpf,
linux-kernel
In-Reply-To: <20190810085821.11cee8b0@lwn.net>
[-- Attachment #1: Type: text/plain, Size: 971 bytes --]
On Sat, Aug 10, 2019 at 08:58:21AM -0600, Jonathan Corbet wrote:
> On Sat, 10 Aug 2019 14:17:37 +0200
> Jonathan Neuschäfer <j.neuschaefer@gmx.net> wrote:
>
> > With the recent change to auto-detect function names, Sphinx parses
> > socket() as a reference to the in-kernel definition of socket. It then
> > decides that struct socket is a good match, which was obviously not
> > intended in this case, because the text speaks about the syscall with
> > the same name.
> >
> > Prevent socket() from being misinterpreted by wrapping it in ``inline
> > literal`` quotes.
> >
> > Signed-off-by: Jonathan Neuschäfer <j.neuschaefer@gmx.net>
>
> Thanks for looking at that. The better fix, though, would be to add
> socket() to the Skipfuncs array in Documentation/sphinx/automarkup.py.
> Then it will do the right thing everywhere without the need to add markup
> to the RST files.
Alright, I'll do that for v2.
Thanks,
Jonathan Neuschäfer
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH] xen-netback: no need to check return value of debugfs_create functions
From: Wei Liu @ 2019-08-11 10:49 UTC (permalink / raw)
To: Greg Kroah-Hartman; +Cc: Wei Liu, Paul Durrant, xen-devel, netdev
In-Reply-To: <20190810103108.GA29487@kroah.com>
On Sat, Aug 10, 2019 at 12:31:08PM +0200, Greg Kroah-Hartman wrote:
> When calling debugfs functions, there is no need to ever check the
> return value. The function can work or not, but the code logic should
> never do something different based on this.
>
> Cc: Wei Liu <wei.liu@kernel.org>
> Cc: Paul Durrant <paul.durrant@citrix.com>
> Cc: xen-devel@lists.xenproject.org
> Cc: netdev@vger.kernel.org
> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Acked-by: Wei Liu <wei.liu@kernel.org>
^ permalink raw reply
* Re: [PATCH net-next] net: openvswitch: Set OvS recirc_id from tc chain index
From: Paul Blakey @ 2019-08-11 10:48 UTC (permalink / raw)
To: Pravin Shelar
Cc: Linux Kernel Network Developers, David S. Miller, Justin Pettit,
Simon Horman, Marcelo Ricardo Leitner, Vlad Buslov, Jiri Pirko,
Roi Dayan, Yossi Kuperman, Rony Efraim, Oz Shlomo
In-Reply-To: <CAOrHB_DhfiQy8RwTiwgn9ZXgsd5j2f0ynZPUP4wf-xzhjwo8kg@mail.gmail.com>
On 8/8/2019 11:53 PM, Pravin Shelar wrote:
> On Wed, Aug 7, 2019 at 5:08 AM Paul Blakey <paulb@mellanox.com> wrote:
>> Offloaded OvS datapath rules are translated one to one to tc rules,
>> for example the following simplified OvS rule:
>>
>> recirc_id(0),in_port(dev1),eth_type(0x0800),ct_state(-trk) actions:ct(),recirc(2)
>>
>> Will be translated to the following tc rule:
>>
>> $ tc filter add dev dev1 ingress \
>> prio 1 chain 0 proto ip \
>> flower tcp ct_state -trk \
>> action ct pipe \
>> action goto chain 2
>>
>> Received packets will first travel though tc, and if they aren't stolen
>> by it, like in the above rule, they will continue to OvS datapath.
>> Since we already did some actions (action ct in this case) which might
>> modify the packets, and updated action stats, we would like to continue
>> the proccessing with the correct recirc_id in OvS (here recirc_id(2))
>> where we left off.
>>
>> To support this, introduce a new skb extension for tc, which
>> will be used for translating tc chain to ovs recirc_id to
>> handle these miss cases. Last tc chain index will be set
>> by tc goto chain action and read by OvS datapath.
>>
>> Signed-off-by: Paul Blakey <paulb@mellanox.com>
>> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
>> Acked-by: Jiri Pirko <jiri@mellanox.com>
>> ---
>> include/linux/skbuff.h | 13 +++++++++++++
>> include/net/sch_generic.h | 5 ++++-
>> net/core/skbuff.c | 6 ++++++
>> net/openvswitch/flow.c | 9 +++++++++
>> net/sched/Kconfig | 13 +++++++++++++
>> net/sched/act_api.c | 1 +
>> net/sched/cls_api.c | 12 ++++++++++++
>> 7 files changed, 58 insertions(+), 1 deletion(-)
>>
>> diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
>> index 3aef8d8..fb2a792 100644
>> --- a/include/linux/skbuff.h
>> +++ b/include/linux/skbuff.h
>> @@ -279,6 +279,16 @@ struct nf_bridge_info {
>> };
>> #endif
>>
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> +/* Chain in tc_skb_ext will be used to share the tc chain with
>> + * ovs recirc_id. It will be set to the current chain by tc
>> + * and read by ovs to recirc_id.
>> + */
>> +struct tc_skb_ext {
>> + __u32 chain;
>> +};
>> +#endif
>> +
>> struct sk_buff_head {
>> /* These two members must be first. */
>> struct sk_buff *next;
>> @@ -4050,6 +4060,9 @@ enum skb_ext_id {
>> #ifdef CONFIG_XFRM
>> SKB_EXT_SEC_PATH,
>> #endif
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + TC_SKB_EXT,
>> +#endif
>> SKB_EXT_NUM, /* must be last */
>> };
>>
>> diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
>> index 6b6b012..871feea 100644
>> --- a/include/net/sch_generic.h
>> +++ b/include/net/sch_generic.h
>> @@ -275,7 +275,10 @@ struct tcf_result {
>> unsigned long class;
>> u32 classid;
>> };
>> - const struct tcf_proto *goto_tp;
>> + struct {
>> + const struct tcf_proto *goto_tp;
>> + u32 goto_index;
>> + };
>>
>> /* used in the skb_tc_reinsert function */
>> struct {
>> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
>> index ea8e8d3..2b40b5a 100644
>> --- a/net/core/skbuff.c
>> +++ b/net/core/skbuff.c
>> @@ -4087,6 +4087,9 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb)
>> #ifdef CONFIG_XFRM
>> [SKB_EXT_SEC_PATH] = SKB_EXT_CHUNKSIZEOF(struct sec_path),
>> #endif
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + [TC_SKB_EXT] = SKB_EXT_CHUNKSIZEOF(struct tc_skb_ext),
>> +#endif
>> };
>>
>> static __always_inline unsigned int skb_ext_total_length(void)
>> @@ -4098,6 +4101,9 @@ static __always_inline unsigned int skb_ext_total_length(void)
>> #ifdef CONFIG_XFRM
>> skb_ext_type_len[SKB_EXT_SEC_PATH] +
>> #endif
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + skb_ext_type_len[TC_SKB_EXT] +
>> +#endif
>> 0;
>> }
>>
>> diff --git a/net/openvswitch/flow.c b/net/openvswitch/flow.c
>> index bc89e16..0287ead 100644
>> --- a/net/openvswitch/flow.c
>> +++ b/net/openvswitch/flow.c
>> @@ -816,6 +816,9 @@ static int key_extract_mac_proto(struct sk_buff *skb)
>> int ovs_flow_key_extract(const struct ip_tunnel_info *tun_info,
>> struct sk_buff *skb, struct sw_flow_key *key)
>> {
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + struct tc_skb_ext *tc_ext;
>> +#endif
>> int res, err;
>>
>> /* Extract metadata from packet. */
>> @@ -848,7 +851,13 @@ int ovs_flow_key_extract(const struct ip_tunnel_info *tun_info,
>> if (res < 0)
>> return res;
>> key->mac_proto = res;
>> +
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + tc_ext = skb_ext_find(skb, TC_SKB_EXT);
>> + key->recirc_id = tc_ext ? tc_ext->chain : 0;
>> +#else
>> key->recirc_id = 0;
>> +#endif
>>
> Most of cases the config would be turned on, so the ifdef is not that
> useful. Can you add static key to avoid searching the skb-ext in non
> offload cases.
Also regarding the ifdefs, yeah , I need them unless I always define the
enum, which other extension don't do.
^ permalink raw reply
* Re: [PATCH net-next] net: openvswitch: Set OvS recirc_id from tc chain index
From: Paul Blakey @ 2019-08-11 10:46 UTC (permalink / raw)
To: Pravin Shelar
Cc: Linux Kernel Network Developers, David S. Miller, Justin Pettit,
Simon Horman, Marcelo Ricardo Leitner, Vlad Buslov, Jiri Pirko,
Roi Dayan, Yossi Kuperman, Rony Efraim, Oz Shlomo
In-Reply-To: <CAOrHB_DhfiQy8RwTiwgn9ZXgsd5j2f0ynZPUP4wf-xzhjwo8kg@mail.gmail.com>
On 8/8/2019 11:53 PM, Pravin Shelar wrote:
> On Wed, Aug 7, 2019 at 5:08 AM Paul Blakey <paulb@mellanox.com> wrote:
>> Offloaded OvS datapath rules are translated one to one to tc rules,
>> for example the following simplified OvS rule:
>>
>> recirc_id(0),in_port(dev1),eth_type(0x0800),ct_state(-trk) actions:ct(),recirc(2)
>>
>> Will be translated to the following tc rule:
>>
>> $ tc filter add dev dev1 ingress \
>> prio 1 chain 0 proto ip \
>> flower tcp ct_state -trk \
>> action ct pipe \
>> action goto chain 2
>>
>> Received packets will first travel though tc, and if they aren't stolen
>> by it, like in the above rule, they will continue to OvS datapath.
>> Since we already did some actions (action ct in this case) which might
>> modify the packets, and updated action stats, we would like to continue
>> the proccessing with the correct recirc_id in OvS (here recirc_id(2))
>> where we left off.
>>
>> To support this, introduce a new skb extension for tc, which
>> will be used for translating tc chain to ovs recirc_id to
>> handle these miss cases. Last tc chain index will be set
>> by tc goto chain action and read by OvS datapath.
>>
>> Signed-off-by: Paul Blakey <paulb@mellanox.com>
>> Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
>> Acked-by: Jiri Pirko <jiri@mellanox.com>
>> ---
>> include/linux/skbuff.h | 13 +++++++++++++
>> include/net/sch_generic.h | 5 ++++-
>> net/core/skbuff.c | 6 ++++++
>> net/openvswitch/flow.c | 9 +++++++++
>> net/sched/Kconfig | 13 +++++++++++++
>> net/sched/act_api.c | 1 +
>> net/sched/cls_api.c | 12 ++++++++++++
>> 7 files changed, 58 insertions(+), 1 deletion(-)
>>
>> diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
>> index 3aef8d8..fb2a792 100644
>> --- a/include/linux/skbuff.h
>> +++ b/include/linux/skbuff.h
>> @@ -279,6 +279,16 @@ struct nf_bridge_info {
>> };
>> #endif
>>
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> +/* Chain in tc_skb_ext will be used to share the tc chain with
>> + * ovs recirc_id. It will be set to the current chain by tc
>> + * and read by ovs to recirc_id.
>> + */
>> +struct tc_skb_ext {
>> + __u32 chain;
>> +};
>> +#endif
>> +
>> struct sk_buff_head {
>> /* These two members must be first. */
>> struct sk_buff *next;
>> @@ -4050,6 +4060,9 @@ enum skb_ext_id {
>> #ifdef CONFIG_XFRM
>> SKB_EXT_SEC_PATH,
>> #endif
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + TC_SKB_EXT,
>> +#endif
>> SKB_EXT_NUM, /* must be last */
>> };
>>
>> diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
>> index 6b6b012..871feea 100644
>> --- a/include/net/sch_generic.h
>> +++ b/include/net/sch_generic.h
>> @@ -275,7 +275,10 @@ struct tcf_result {
>> unsigned long class;
>> u32 classid;
>> };
>> - const struct tcf_proto *goto_tp;
>> + struct {
>> + const struct tcf_proto *goto_tp;
>> + u32 goto_index;
>> + };
>>
>> /* used in the skb_tc_reinsert function */
>> struct {
>> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
>> index ea8e8d3..2b40b5a 100644
>> --- a/net/core/skbuff.c
>> +++ b/net/core/skbuff.c
>> @@ -4087,6 +4087,9 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb)
>> #ifdef CONFIG_XFRM
>> [SKB_EXT_SEC_PATH] = SKB_EXT_CHUNKSIZEOF(struct sec_path),
>> #endif
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + [TC_SKB_EXT] = SKB_EXT_CHUNKSIZEOF(struct tc_skb_ext),
>> +#endif
>> };
>>
>> static __always_inline unsigned int skb_ext_total_length(void)
>> @@ -4098,6 +4101,9 @@ static __always_inline unsigned int skb_ext_total_length(void)
>> #ifdef CONFIG_XFRM
>> skb_ext_type_len[SKB_EXT_SEC_PATH] +
>> #endif
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + skb_ext_type_len[TC_SKB_EXT] +
>> +#endif
>> 0;
>> }
>>
>> diff --git a/net/openvswitch/flow.c b/net/openvswitch/flow.c
>> index bc89e16..0287ead 100644
>> --- a/net/openvswitch/flow.c
>> +++ b/net/openvswitch/flow.c
>> @@ -816,6 +816,9 @@ static int key_extract_mac_proto(struct sk_buff *skb)
>> int ovs_flow_key_extract(const struct ip_tunnel_info *tun_info,
>> struct sk_buff *skb, struct sw_flow_key *key)
>> {
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + struct tc_skb_ext *tc_ext;
>> +#endif
>> int res, err;
>>
>> /* Extract metadata from packet. */
>> @@ -848,7 +851,13 @@ int ovs_flow_key_extract(const struct ip_tunnel_info *tun_info,
>> if (res < 0)
>> return res;
>> key->mac_proto = res;
>> +
>> +#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
>> + tc_ext = skb_ext_find(skb, TC_SKB_EXT);
>> + key->recirc_id = tc_ext ? tc_ext->chain : 0;
>> +#else
>> key->recirc_id = 0;
>> +#endif
>>
> Most of cases the config would be turned on, so the ifdef is not that
> useful. Can you add static key to avoid searching the skb-ext in non
> offload cases.
Hi,
What do you mean by a static key?
^ permalink raw reply
* Re: [PATCH 11/11] treewide: remove dummy Makefiles for single targets
From: Leon Romanovsky @ 2019-08-11 9:55 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild@vger.kernel.org, Christoph Hellwig, Sam Ravnborg,
Alexei Starovoitov, Boris Pismenny, Daniel Borkmann,
David S. Miller, Igor Russkikh, Jakub Kicinski, Martin KaFai Lau,
Saeed Mahameed, Song Liu, Yonghong Song, bpf@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org,
netdev@vger.kernel.org, oss-drivers@netronome.com
In-Reply-To: <20190810155307.29322-12-yamada.masahiro@socionext.com>
On Sun, Aug 11, 2019 at 12:53:07AM +0900, Masahiro Yamada wrote:
> Now that the single target build descends into sub-directories
> in the same ways as the normal build, these dummy Makefiles
> are not needed any more.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
It is hard to test/review/ack on this patch without seeing previous
patches, especially patch #10 where you changed logic of single targets.
Thanks
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox