* [PATCH v2] ipv4: make DSCP values works with ip rules
From: Pavel Balaev @ 2018-11-20 13:29 UTC (permalink / raw)
To: davem; +Cc: netdev
This patch adds ability to set DSCP values in ip rules.
Values presented in /etc/iproute3/rt_dsfield and now can be used in rules.
Example:
$ ip ru add from 10.88.0.2 tos AF23 lookup dscp.
Result:
---
32762: from 10.88.0.2 tos AF43 lookup dscp
---
Patch was tested with such configuration:
+-----------+
| 10.88.0.2 | -> ping 172.16.0.1 -Q 0x58 ->
| host0 |
+-----------+
+-------------------------------------+
| router with patched kernel |
| |
-> | from 10.88.0.2 tos AF43 lookup dscp |->
| table dscp: |
| 172.16.0.0/24 via 10.200.0.2 |
+-------------------------------------+
+------------------+
-> | eth0: 10.200.0.2 |
| eth1: 172.16.0.1 |
| host1 |
+------------------+
Signed-off-by: Pavel Balaev <mail@void.so>
---
include/net/route.h | 2 +-
include/uapi/linux/ip.h | 2 ++
net/ipv4/fib_rules.c | 2 +-
3 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/include/net/route.h b/include/net/route.h
index bb53cdba..b984ecff 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -239,7 +239,7 @@ static inline void ip_rt_put(struct rtable *rt)
dst_release(&rt->dst);
}
-#define IPTOS_RT_MASK (IPTOS_TOS_MASK & ~3)
+#define IPTOS_RT_MASK (IPTOS_DSCP_MASK & ~3)
extern const __u8 ip_tos2prio[16];
diff --git a/include/uapi/linux/ip.h b/include/uapi/linux/ip.h
index e42d13b5..307ce2b1 100644
--- a/include/uapi/linux/ip.h
+++ b/include/uapi/linux/ip.h
@@ -38,6 +38,8 @@
#define IPTOS_PREC_PRIORITY 0x20
#define IPTOS_PREC_ROUTINE 0x00
+#define IPTOS_DSCP_MASK (IPTOS_TOS_MASK | IPTOS_PREC_MASK)
+#define IPTOS_DSCP(tos) ((tos)&IPTOS_DSCP_MASK)
/* IP options */
#define IPOPT_COPY 0x80
diff --git a/net/ipv4/fib_rules.c b/net/ipv4/fib_rules.c
index f8eb78d0..9ba91ef0 100644
--- a/net/ipv4/fib_rules.c
+++ b/net/ipv4/fib_rules.c
@@ -220,7 +220,7 @@ static int fib4_rule_configure(struct fib_rule *rule, struct sk_buff *skb,
int err = -EINVAL;
struct fib4_rule *rule4 = (struct fib4_rule *) rule;
- if (frh->tos & ~IPTOS_TOS_MASK) {
+ if (frh->tos & ~IPTOS_DSCP_MASK) {
NL_SET_ERR_MSG(extack, "Invalid tos");
goto errout;
}
--
2.18.1
^ permalink raw reply related
* Re: [PATCH net] sctp: hold transport before accessing its asoc in sctp_hash_transport
From: Neil Horman @ 2018-11-20 12:52 UTC (permalink / raw)
To: Xin Long; +Cc: network dev, linux-sctp, davem, Marcelo Ricardo Leitner
In-Reply-To: <b1ff91726b33df038aa57cbb8103dee29436d349.1542712156.git.lucien.xin@gmail.com>
On Tue, Nov 20, 2018 at 07:09:16PM +0800, Xin Long wrote:
> In sctp_hash_transport, it dereferences a transport's asoc only under
> rcu_read_lock. Without holding the transport, its asoc could be freed
> already, which leads to a use-after-free panic.
>
> A similar fix as Commit bab1be79a516 ("sctp: hold transport before
> accessing its asoc in sctp_transport_get_next") is needed to hold
> the transport before accessing its asoc in sctp_hash_transport.
>
> Fixes: cd2b70875058 ("sctp: check duplicate node before inserting a new transport")
> Reported-by: syzbot+0b05d8aa7cb185107483@syzkaller.appspotmail.com
> Signed-off-by: Xin Long <lucien.xin@gmail.com>
> ---
> net/sctp/input.c | 7 ++++++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/net/sctp/input.c b/net/sctp/input.c
> index 5c36a99..69584e9 100644
> --- a/net/sctp/input.c
> +++ b/net/sctp/input.c
> @@ -896,11 +896,16 @@ int sctp_hash_transport(struct sctp_transport *t)
> list = rhltable_lookup(&sctp_transport_hashtable, &arg,
> sctp_hash_params);
>
> - rhl_for_each_entry_rcu(transport, tmp, list, node)
> + rhl_for_each_entry_rcu(transport, tmp, list, node) {
> + if (!sctp_transport_hold(transport))
> + continue;
> if (transport->asoc->ep == t->asoc->ep) {
> + sctp_transport_put(transport);
> rcu_read_unlock();
> return -EEXIST;
> }
> + sctp_transport_put(transport);
> + }
> rcu_read_unlock();
>
> err = rhltable_insert_key(&sctp_transport_hashtable, &arg,
> --
> 2.1.0
>
>
something doesn't feel at all right about this. If we are inserting a transport
to an association, it would seem to me that we should have at least one user of
the association (i.e. non-zero refcount). As such it seems something is wrong
with the association refcount here. At the very least, if there is a case where
an association is being removed while a transport is being added, the better
solution would be to ensure that sctp_association_destroy goes through a
quiescent point prior to unhashing transports from the list, to ensure that
there is no conflict with the add operation above.
Neil
^ permalink raw reply
* [PATCH net-next] net: bcmgenet: remove HFB_CTRL access
From: Doug Berger @ 2018-11-20 23:17 UTC (permalink / raw)
To: David S. Miller; +Cc: Florian Fainelli, netdev, linux-kernel, Doug Berger
Commit c5a54bbcecec ("net: bcmgenet: abort suspend on error")
mistakenly introduced register accesses that should not occur
in bcmgenet_wol_power_up_cfg().
Fixes: c5a54bbcecec ("net: bcmgenet: abort suspend on error")
Signed-off-by: Doug Berger <opendmb@gmail.com>
---
drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c | 4 ----
1 file changed, 4 deletions(-)
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
index b3596e0ee47b..57582efa362d 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
@@ -191,10 +191,6 @@ void bcmgenet_wol_power_up_cfg(struct bcmgenet_priv *priv,
reg &= ~MPD_EN;
bcmgenet_umac_writel(priv, reg, UMAC_MPD_CTRL);
- reg = bcmgenet_hfb_reg_readl(priv, HFB_CTRL);
- reg &= ~(RBUF_HFB_EN | RBUF_ACPI_EN);
- bcmgenet_hfb_reg_writel(priv, reg, HFB_CTRL);
-
/* Disable CRC Forward */
reg = bcmgenet_umac_readl(priv, UMAC_CMD);
reg &= ~CMD_CRC_FWD;
--
2.7.4
^ permalink raw reply related
* [PATCH net-next 4/4] mlx5: use skb_vlan_tag_get_prio()
From: Michał Mirosław @ 2018-11-20 12:20 UTC (permalink / raw)
To: netdev
Cc: Saeed Mahameed, Leon Romanovsky, linux-rdma, Ajit Khaparde, devel,
Haiyang Zhang, K. Y. Srinivasan, Sathya Perla, Somnath Kotur,
Sriharsha Basavapatna, Stephen Hemminger
In-Reply-To: <cover.1542716156.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 6dacaeba2fbf..9afdf955f2bc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -127,7 +127,7 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
else
#endif
if (skb_vlan_tag_present(skb))
- up = skb->vlan_tci >> VLAN_PRIO_SHIFT;
+ up = skb_vlan_tag_get_prio(skb);
/* channel_ix can be larger than num_channels since
* dev->num_real_tx_queues = num_channels * num_tc
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 3/4] benet: use skb_vlan_tag_get_prio()
From: Michał Mirosław @ 2018-11-20 12:20 UTC (permalink / raw)
To: netdev
Cc: Sathya Perla, Ajit Khaparde, Sriharsha Basavapatna, Somnath Kotur,
devel, Haiyang Zhang, K. Y. Srinivasan, Leon Romanovsky,
linux-rdma, Saeed Mahameed, Stephen Hemminger
In-Reply-To: <cover.1542716156.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
drivers/net/ethernet/emulex/benet/be_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/emulex/benet/be_main.c b/drivers/net/ethernet/emulex/benet/be_main.c
index 80b2bd3747ce..245abf0d19c0 100644
--- a/drivers/net/ethernet/emulex/benet/be_main.c
+++ b/drivers/net/ethernet/emulex/benet/be_main.c
@@ -796,7 +796,7 @@ static inline u16 be_get_tx_vlan_tag(struct be_adapter *adapter,
u16 vlan_tag;
vlan_tag = skb_vlan_tag_get(skb);
- vlan_prio = (vlan_tag & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT;
+ vlan_prio = skb_vlan_tag_get_prio(skb);
/* If vlan priority provided by OS is NOT in available bmap */
if (!(adapter->vlan_prio_bmap & (1 << vlan_prio)))
vlan_tag = (vlan_tag & ~VLAN_PRIO_MASK) |
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 2/4] net/hyperv: use skb_vlan_tag_*() helpers
From: Michał Mirosław @ 2018-11-20 12:20 UTC (permalink / raw)
To: netdev
Cc: K. Y. Srinivasan, Haiyang Zhang, Stephen Hemminger, devel,
Ajit Khaparde, Leon Romanovsky, linux-rdma, Saeed Mahameed,
Sathya Perla, Somnath Kotur, Sriharsha Basavapatna
In-Reply-To: <cover.1542716156.git.mirq-linux@rere.qmqm.pl>
Replace open-coded bitfield manipulation with skb_vlan_tag_*() helpers.
This also enables correctly passing of VLAN.CFI bit.
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
drivers/net/hyperv/netvsc_drv.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index cf36e7ff3191..85936ed9e952 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -605,9 +605,9 @@ static int netvsc_start_xmit(struct sk_buff *skb, struct net_device *net)
IEEE_8021Q_INFO);
vlan->value = 0;
- vlan->vlanid = skb->vlan_tci & VLAN_VID_MASK;
- vlan->pri = (skb->vlan_tci & VLAN_PRIO_MASK) >>
- VLAN_PRIO_SHIFT;
+ vlan->vlanid = skb_vlan_tag_get_id(skb);
+ vlan->cfi = skb_vlan_tag_get_cfi(skb);
+ vlan->pri = skb_vlan_tag_get_prio(skb);
}
if (skb_is_gso(skb)) {
@@ -781,7 +781,8 @@ static struct sk_buff *netvsc_alloc_recv_skb(struct net_device *net,
}
if (vlan) {
- u16 vlan_tci = vlan->vlanid | (vlan->pri << VLAN_PRIO_SHIFT);
+ u16 vlan_tci = vlan->vlanid | (vlan->pri << VLAN_PRIO_SHIFT) |
+ (vlan->cfi ? VLAN_CFI_MASK : 0);
__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q),
vlan_tci);
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 0/4] VLAN tag handling cleanup
From: Michał Mirosław @ 2018-11-20 12:20 UTC (permalink / raw)
To: netdev
Cc: Ajit Khaparde, devel, Haiyang Zhang, K. Y. Srinivasan,
Leon Romanovsky, linux-rdma, Saeed Mahameed, Sathya Perla,
Somnath Kotur, Sriharsha Basavapatna, Stephen Hemminger
This is a cleanup set after VLAN_TAG_PRESENT removal. The CFI bit
handling is made similar to how other tag fields are used.
Michał Mirosław (4):
net/vlan: introduce skb_vlan_tag_get_cfi() helper
net/hyperv: use skb_vlan_tag_*() helpers
benet: use skb_vlan_tag_get_prio()
mlx5: use skb_vlan_tag_get_prio()
drivers/net/ethernet/emulex/benet/be_main.c | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 2 +-
drivers/net/hyperv/netvsc_drv.c | 9 +++++----
include/linux/if_vlan.h | 3 ++-
4 files changed, 9 insertions(+), 7 deletions(-)
--
2.19.1
^ permalink raw reply
* [PATCH net-next 1/4] net/vlan: introduce skb_vlan_tag_get_cfi() helper
From: Michał Mirosław @ 2018-11-20 12:20 UTC (permalink / raw)
To: netdev
Cc: Ajit Khaparde, devel, Haiyang Zhang, K. Y. Srinivasan,
Leon Romanovsky, linux-rdma, Saeed Mahameed, Sathya Perla,
Somnath Kotur, Sriharsha Basavapatna, Stephen Hemminger
In-Reply-To: <cover.1542716156.git.mirq-linux@rere.qmqm.pl>
Abstract CFI/DEI bit access consistently with other VLAN tag fields.
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
include/linux/if_vlan.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/include/linux/if_vlan.h b/include/linux/if_vlan.h
index 7a541eadf78e..4cca4da7a6de 100644
--- a/include/linux/if_vlan.h
+++ b/include/linux/if_vlan.h
@@ -65,7 +65,7 @@ static inline struct vlan_ethhdr *vlan_eth_hdr(const struct sk_buff *skb)
#define VLAN_PRIO_MASK 0xe000 /* Priority Code Point */
#define VLAN_PRIO_SHIFT 13
-#define VLAN_CFI_MASK 0x1000 /* Canonical Format Indicator */
+#define VLAN_CFI_MASK 0x1000 /* Canonical Format Indicator / Drop Eligible Indicator */
#define VLAN_VID_MASK 0x0fff /* VLAN Identifier */
#define VLAN_N_VID 4096
@@ -80,6 +80,7 @@ static inline bool is_vlan_dev(const struct net_device *dev)
#define skb_vlan_tag_present(__skb) ((__skb)->vlan_present)
#define skb_vlan_tag_get(__skb) ((__skb)->vlan_tci)
#define skb_vlan_tag_get_id(__skb) ((__skb)->vlan_tci & VLAN_VID_MASK)
+#define skb_vlan_tag_get_cfi(__skb) (!!((__skb)->vlan_tci & VLAN_CFI_MASK))
#define skb_vlan_tag_get_prio(__skb) (((__skb)->vlan_tci & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT)
static inline int vlan_get_rx_ctag_filter_info(struct net_device *dev)
--
2.19.1
^ permalink raw reply related
* Re: [RFCv3 PATCH 1/6] uacce: Add documents for WarpDrive/uacce
From: Jean-Philippe Brucker @ 2018-11-20 12:19 UTC (permalink / raw)
To: Jonathan Cameron, Jason Gunthorpe
Cc: liguozhu@hisilicon.com, Leon Romanovsky, Kenneth Lee, Tim Sell,
linux-doc@vger.kernel.org, Alexander Shishkin, Zaibo Xu,
zhangfei.gao@foxmail.com, linuxarm@huawei.com,
haojian.zhuang@linaro.org, Christoph Lameter, Hao Fang,
Gavin Schenk, RDMA mailing list, Zhou Wang, Doug Ledford,
Uwe Kleine-König
In-Reply-To: <20181120091650.0000419a@huawei.com>
On 20/11/2018 09:16, Jonathan Cameron wrote:
> +CC Jean-Phillipe and iommu list.
Thanks for the Cc, sorry I don't have enough bandwidth to follow this
thread at the moment.
>>>>> In WarpDrive/uacce, we make this simple. If you support IOMMU and it support
>>>>> SVM/SVA. Everything will be fine just like ODP implicit mode. And you don't need
>>>>> to write any code for that. Because it has been done by IOMMU framework. If it
>>>>
>>>> Looks like the IOMMU code uses mmu_notifier, so it is identical to
>>>> IB's ODP. The only difference is that IB tends to have the IOMMU page
>>>> table in the device, not in the CPU.
>>>>
>>>> The only case I know if that is different is the new-fangled CAPI
>>>> stuff where the IOMMU can directly use the CPU's page table and the
>>>> IOMMU page table (in device or CPU) is eliminated.
>>>
>>> Yes. We are not focusing on the current implementation. As mentioned in the
>>> cover letter. We are expecting Jean Philips' SVA patch:
>>> git://linux-arm.org/linux-jpb.
>>
>> This SVA stuff does not look comparable to CAPI as it still requires
>> maintaining seperate IOMMU page tables.
With SVA, we use the same page tables in the IOMMU and CPU. It's the
same pgd pointer, there is no mirroring of mappings. We bind the process
page tables with the device using a PASID (Process Address Space ID).
After fork(), the child's mm is different from the parent's one, and is
not automatically bound to the device. The device driver will have to
issue a new bind() request, and the child mm will be bound with a
different PASID.
There could be a problem if the child inherits the parent's device
handle. Then depending on the device, the child could be able to program
DMA and possibly access the parent's address space. The parent needs to
be aware of that when using the bind() API, and close the device fd in
the child after fork().
We use MMU notifiers for some address space changes:
* The IOTLB needs to be invalidated after any unmap() to the process
address space. On Arm systems the SMMU IOTLBs can be invalidated by the
CPU TLBI instructions, but we still need to invalidate TLBs private to
devices that are arch-agnostic (Address Translation Cache in PCI ATS).
* When the process mm exits, we need to remove the associated PASID
configuration in the IOMMU and invalidate the TLBs.
Thanks,
Jean
^ permalink raw reply
* Geschäftsvorschlag
From: Mr. Alan Austin @ 2018-11-20 11:52 UTC (permalink / raw)
To: Recipients
Hallo,
Ich bin Mr. Alan Austin und arbeite derzeit mit der Credit suisse Bank London. Ich habe Ihren Kontakt während meiner privaten Suche gesehen und bin der festen Überzeugung, dass Sie sehr ehrlich, engagiert und in der Lage sein werden, dieses Unternehmen zu unterstützen.
Ich bin ein Kontoführer des verstorbenen Dr. Manzoor Hassan, der mit seiner gesamten Familie in Syrien gestorben ist. Darauf basierend kontaktiere ich Sie, um meinem verstorbenen Klienten als Begünstigter zur Verfügung zu stehen, damit sein Geld in unserem Sorgerecht freigelassen wird Ihnen als Begünstigter an den Verstorbenen gezahlt.
Es ist wichtig, dass Sie mir mit Ihrem vollständigen Namen und Ihrer Adresse antworten, einschließlich Ihrer direkten Telefonnummer, damit ich Ihnen vollständige Details dieser Transaktion und weitere Informationen über meinen verstorbenen Kunden geben kann, der einen riesigen Geldbetrag in unserer Bank hinterlassen hat. Ich werde Ihnen alle erforderlichen Informationen, Unterlagen und Nachweise zur Verfügung stellen, um die Forderung der verschiedenen Ämter rechtlich zu sichern, damit der Fonds als wahrer Begünstigter auf ein beliebiges Konto überwiesen werden kann.
Dein,
Herr Alan Austin
^ permalink raw reply
* Re: Realtek NIC uses over 1 Watt with no traffic
From: Heiner Kallweit @ 2018-11-20 22:29 UTC (permalink / raw)
To: Paul Menzel, Andrew Lunn
Cc: Realtek linux nic maintainers, David S. Miller, netdev,
linux-kernel
In-Reply-To: <f75d832e-4b5b-8c7c-d5d9-013f9d5cacef@molgen.mpg.de>
On 20.11.2018 23:25, Paul Menzel wrote:
> Dear Heiner,
>
>
> Am 20.11.18 um 22:06 schrieb Heiner Kallweit:
>> On 20.11.2018 21:31, Paul Menzel wrote:
>
> […]
>
>>> Am 20.11.18 um 21:14 schrieb Heiner Kallweit:
>>>> On 20.11.2018 15:45, Andrew Lunn wrote:
>>>>> On Tue, Nov 20, 2018 at 09:40:25AM +0100, Paul Menzel wrote:
>>>
>>>>>> Using Ubuntu 18.10, Linux 4.18.0-11-generic, PowerTOP 2.9 shows, the NIC
>>>>>> uses 1.77 Watts. A network cable is plugged in, but there is no real traffic
>>>>>> according to `iftop`. Only an email program is running.
>>>>>>
>>>>>> $ lspci -nn -s 3:00.1
>>>>>> 03:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
>>>>>> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev
>>>>>> 12)
>>>>>>
>>>>>> Is that a measurement error, or does the NIC really need that much power?
>>>
>>>>> This sounds like Energy Efficient Ethernet, EEE, is not enabled.
>>>>>
>>>>> What does ethtool --show-eee ethX say?
>>>
>>> $ sudo ethtool --show-eee enp3s0f1
>>> Cannot get EEE settings: Operation not supported
>>>
>>>> The r8169 driver doesn't support the get_eee ethtool_ops callback.
>>>> For certain chip versions EEE gets enabled in the PHY init, for others
>>>> not and some don't seem to support EEE at all.
>>>>
>>>> Apart from EEE one important factor affecting power consumption is ASPM.
>>>> This was recently enabled for certain chip versions.
>>>>
>>>> Information that would help:
>>>>
>>>> whether Wake-on-LAN is enabled ("Wake-on:" line from ethtool output)
>>>
>>> ```
>>> $ sudo ethtool enp3s0f1
>>> Settings for enp3s0f1:
>>> Supported ports: [ TP AUI BNC MII FIBRE ]
>>> Supported link modes: 10baseT/Half 10baseT/Full
>>> 100baseT/Half 100baseT/Full
>>> 1000baseT/Full
>>> Supported pause frame use: Symmetric Receive-only
>>> Supports auto-negotiation: Yes
>>> Supported FEC modes: Not reported
>>> Advertised link modes: 10baseT/Half 10baseT/Full
>>> 100baseT/Half 100baseT/Full
>>> 1000baseT/Full
>>> Advertised pause frame use: Symmetric Receive-only
>>> Advertised auto-negotiation: Yes
>>> Advertised FEC modes: Not reported
>>> Link partner advertised link modes: 10baseT/Half 10baseT/Full
>>> 100baseT/Half 100baseT/Full
>>> 1000baseT/Full
>>> Link partner advertised pause frame use: Symmetric
>>> Link partner advertised auto-negotiation: Yes
>>> Link partner advertised FEC modes: Not reported
>>> Speed: 1000Mb/s
>>> Duplex: Full
>>> Port: MII
>>> PHYAD: 0
>>> Transceiver: internal
>>> Auto-negotiation: on
>>> Supports Wake-on: pumbg
>>> Wake-on: g
>>> Current message level: 0x00000033 (51)
>>> drv probe ifdown ifup
>>> Link detected: yes
>>> ```
>>>
>>> So, it’s enabled (g Wake on MagicPacket(tm)).
>>>
>>> Running `sudo ethtool -s enp3s0f1 wol d;` doesn’t change anything though.
>>>
>>>> lspci -vv output for the Realtek NIC
>>>
>>> Here is the output (quoted, so that Thunderbird does not wrap the line).
>>>
>>>> $ sudo lspci -vv -s 3:00.1
>>>> 03:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 12)
>>>> Subsystem: CLEVO/KAPOK Computer RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>>>> Latency: 0, Cache Line Size: 64 bytes
>>>> Interrupt: pin A routed to IRQ 19
>>>> Region 0: I/O ports at e000 [size=256]
>>>> Region 2: Memory at df114000 (64-bit, non-prefetchable) [size=4K]
>>>> Region 4: Memory at df110000 (64-bit, non-prefetchable) [size=16K]
>>>> Capabilities: [40] Power Management version 3
>>>> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>>> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>>>> Address: 0000000000000000 Data: 0000
>>>> Capabilities: [70] Express (v2) Endpoint, MSI 01
>>>> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
>>>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
>>>> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>>>> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
>>>> MaxPayload 128 bytes, MaxReadReq 4096 bytes
>>>> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
>>>> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
>>>> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>>>> LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
>>
>> L0s is missing here, no idea why.
>
> Indeed. I’ll forward that to TUXEDO.
>
>>>> ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
>>>> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>>>> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE#
>>>> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
>>>> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
>>>> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>>>> Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
>>>> Vector table: BAR=4 offset=00000000
>>>> PBA: BAR=4 offset=00000800
>>>> Capabilities: [d0] Vital Product Data
>>>> pcilib: sysfs_read_vpd: read failed: Input/output error
>>>> Not readable
>>>> Capabilities: [100 v2] Advanced Error Reporting
>>>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>>> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>>>> CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ NonFatalErr+
>>>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>>>> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>>>> Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
>>>> Capabilities: [170 v1] Latency Tolerance Reporting
>>>> Max snoop latency: 3145728ns
>>>> Max no snoop latency: 3145728ns
>>>> Capabilities: [178 v1] L1 PM Substates
>>>> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
>>>> PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
>>>> L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
>>>> T_CommonMode=0us LTR1.2_Threshold=0ns
>>>> L1SubCtl2: T_PwrOn=10us
>>>> Kernel driver in use: r8169
>>>> Kernel modules: r8169
>>>
>>> Some Active State Power Management levels seem to be enabled.
>>>
>>>> Info from powertop about package C states. With ASPM my system reaches
>>>> 50% PC7 + 50% PC10.
>>>
>>> That seems to be the case on my TUXEDO Book BU1406 too.
>>>
>>>> Paket | Kern | CPU 0 CPU 2
>>>> | | C0 aktiv 1,7% 1,1%
>>>> | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
>>>> | | C1E 0,2% 0,8 ms 0,1% 0,2 ms
>>>> C2 (pc2) 5,2% | |
>>>> C3 (pc3) 82,1% | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,1% 0,2 ms
>>
>> Relevant are the package states and your system reaches pc3 only. The "Tunables" section
>> in powertop may provide hints how to save more power.
>
> Thank you for the hint. As it’s unrelated, I’ll just paste the tunables below, but will try to forward it to the correct people.
>
> Schlecht Audiocodec-Energieverwaltung einschalten
> Schlecht VM-Rückschreibezeitlimit
>
>>>> C6 (pc6) 0,0% | C6 (cc6) 1,3% | C6 0,8% 0,5 ms 1,4% 0,6 ms
>>>> C7 (pc7) 0,0% | C7 (cc7) 90,8% | C7s 0,0% 1,6 ms 0,0% 0,0 ms
>>>> C8 (pc8) 0,0% | | C8 6,0% 1,8 ms 10,1% 2,0 ms
>>>> C9 (pc9) 0,0% | | C9 0,2% 2,8 ms 0,2% 2,9 ms
>>>> C10 (pc10) 0,0% | | C10 88,7% 12,7 ms 84,4% 14,9 ms
>>>>
>>>> | Kern | CPU 1 CPU 3
>>>> | | C0 aktiv 1,0% 0,8%
>>>> | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
>>>> | | C1E 0,1% 0,3 ms 0,1% 0,3 ms
>>>> | |
>>>> | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,0% 0,2 ms
>>>> | C6 (cc6) 1,1% | C6 0,9% 0,6 ms 0,8% 0,5 ms
>>>> | C7 (cc7) 92,2% | C7s 0,0% 1,7 ms 0,0% 0,0 ms
>>>> | | C8 6,2% 1,7 ms 5,4% 1,7 ms
>>>> | | C9 0,3% 1,7 ms 0,1% 1,9 ms
>>>> | | C10 88,8% 12,1 ms 90,7% 14,8 ms
>>>>
>>>> | GPU |
>>>> | |
>>>> | Powered On 2,2% |
>>>> | RC6 97,8% |
>>>> | RC6p 0,0% |
>>>> | RC6pp 0,0% |
>>>
>>>> dmesg output filtered for "r8169". Primarily relevant is the line with
>>>> the chip name and XID.
>>>
>>> Please find them below.
>>>
>>>> $ sudo dmesg | grep r8169
>>>> [ 5.318442] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 418
>>>> [ 5.318470] r8169 0000:03:00.1: enabling device (0000 -> 0003)
>>>> [ 5.340324] libphy: r8169: probed
>>>> [ 5.340630] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 136
>>
>> Good to know. For this chip version rtl8168g_2_hw_phy_config() is used to configure the PHY,
>> but this function just loads the firmware. So we don't know whether EEE is enabled.
>>
>> What you could do to test further is limiting the speed to 100MBit or 10MBit via ethtool.
>> If this reduces power consumption significantly it's a hint that indeed the PHY seems
>> to be the one to be blamed.
>
> With `sudo ethtool -s enp3s0f1 speed 10 duplex full` the power usage drops to 800 mW and even to 0, so it’s much less as with 1 Gbit/s.
>
OK, so Andrew was right and the issue seems to be the disabled EEE.
I'll set this on my agenda. Most likely in step 1 you'll have to use
ethtool to switch on EEE, in step 2 EEE will be enabled per default
for this chip version.
>>>> [ 5.340632] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
>>>> [ 5.340673] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 9217 usecs
>>>> [ 5.799967] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
>>>> [ 10.036968] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
>>>> [ 676.940934] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 22235
>>>> [ 676.952411] libphy: r8169: probed
>>>> [ 676.952701] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 139
>>>> [ 676.952702] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
>>>> [ 676.952736] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 11518 usecs
>>>> [ 676.954420] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
>>>> [ 676.975161] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
>>>> [ 680.518923] r8169 0000:03:00.1 enp3s0f1: Link is Up - 1Gbps/Full - flow control rx/tx
>>>> [ 1751.285899] r8169 0000:03:00.1: invalid short VPD tag 00 at offset 1
>
>
> Kind regards,
>
> Paul
>
^ permalink raw reply
* Re: Realtek NIC uses over 1 Watt with no traffic
From: Paul Menzel @ 2018-11-20 22:25 UTC (permalink / raw)
To: Heiner Kallweit, Andrew Lunn
Cc: Realtek linux nic maintainers, David S. Miller, netdev,
linux-kernel
In-Reply-To: <25b108c7-342e-e123-eb35-d1e6823102ea@gmail.com>
Dear Heiner,
Am 20.11.18 um 22:06 schrieb Heiner Kallweit:
> On 20.11.2018 21:31, Paul Menzel wrote:
[…]
>> Am 20.11.18 um 21:14 schrieb Heiner Kallweit:
>>> On 20.11.2018 15:45, Andrew Lunn wrote:
>>>> On Tue, Nov 20, 2018 at 09:40:25AM +0100, Paul Menzel wrote:
>>
>>>>> Using Ubuntu 18.10, Linux 4.18.0-11-generic, PowerTOP 2.9 shows, the NIC
>>>>> uses 1.77 Watts. A network cable is plugged in, but there is no real traffic
>>>>> according to `iftop`. Only an email program is running.
>>>>>
>>>>> $ lspci -nn -s 3:00.1
>>>>> 03:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
>>>>> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev
>>>>> 12)
>>>>>
>>>>> Is that a measurement error, or does the NIC really need that much power?
>>
>>>> This sounds like Energy Efficient Ethernet, EEE, is not enabled.
>>>>
>>>> What does ethtool --show-eee ethX say?
>>
>> $ sudo ethtool --show-eee enp3s0f1
>> Cannot get EEE settings: Operation not supported
>>
>>> The r8169 driver doesn't support the get_eee ethtool_ops callback.
>>> For certain chip versions EEE gets enabled in the PHY init, for others
>>> not and some don't seem to support EEE at all.
>>>
>>> Apart from EEE one important factor affecting power consumption is ASPM.
>>> This was recently enabled for certain chip versions.
>>>
>>> Information that would help:
>>>
>>> whether Wake-on-LAN is enabled ("Wake-on:" line from ethtool output)
>>
>> ```
>> $ sudo ethtool enp3s0f1
>> Settings for enp3s0f1:
>> Supported ports: [ TP AUI BNC MII FIBRE ]
>> Supported link modes: 10baseT/Half 10baseT/Full
>> 100baseT/Half 100baseT/Full
>> 1000baseT/Full
>> Supported pause frame use: Symmetric Receive-only
>> Supports auto-negotiation: Yes
>> Supported FEC modes: Not reported
>> Advertised link modes: 10baseT/Half 10baseT/Full
>> 100baseT/Half 100baseT/Full
>> 1000baseT/Full
>> Advertised pause frame use: Symmetric Receive-only
>> Advertised auto-negotiation: Yes
>> Advertised FEC modes: Not reported
>> Link partner advertised link modes: 10baseT/Half 10baseT/Full
>> 100baseT/Half 100baseT/Full
>> 1000baseT/Full
>> Link partner advertised pause frame use: Symmetric
>> Link partner advertised auto-negotiation: Yes
>> Link partner advertised FEC modes: Not reported
>> Speed: 1000Mb/s
>> Duplex: Full
>> Port: MII
>> PHYAD: 0
>> Transceiver: internal
>> Auto-negotiation: on
>> Supports Wake-on: pumbg
>> Wake-on: g
>> Current message level: 0x00000033 (51)
>> drv probe ifdown ifup
>> Link detected: yes
>> ```
>>
>> So, it’s enabled (g Wake on MagicPacket(tm)).
>>
>> Running `sudo ethtool -s enp3s0f1 wol d;` doesn’t change anything though.
>>
>>> lspci -vv output for the Realtek NIC
>>
>> Here is the output (quoted, so that Thunderbird does not wrap the line).
>>
>>> $ sudo lspci -vv -s 3:00.1
>>> 03:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 12)
>>> Subsystem: CLEVO/KAPOK Computer RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>>> Latency: 0, Cache Line Size: 64 bytes
>>> Interrupt: pin A routed to IRQ 19
>>> Region 0: I/O ports at e000 [size=256]
>>> Region 2: Memory at df114000 (64-bit, non-prefetchable) [size=4K]
>>> Region 4: Memory at df110000 (64-bit, non-prefetchable) [size=16K]
>>> Capabilities: [40] Power Management version 3
>>> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>>> Address: 0000000000000000 Data: 0000
>>> Capabilities: [70] Express (v2) Endpoint, MSI 01
>>> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
>>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
>>> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>>> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
>>> MaxPayload 128 bytes, MaxReadReq 4096 bytes
>>> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
>>> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
>>> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>>> LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
>
> L0s is missing here, no idea why.
Indeed. I’ll forward that to TUXEDO.
>>> ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
>>> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>>> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE#
>>> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
>>> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
>>> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>>> Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
>>> Vector table: BAR=4 offset=00000000
>>> PBA: BAR=4 offset=00000800
>>> Capabilities: [d0] Vital Product Data
>>> pcilib: sysfs_read_vpd: read failed: Input/output error
>>> Not readable
>>> Capabilities: [100 v2] Advanced Error Reporting
>>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>>> CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ NonFatalErr+
>>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>>> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>>> Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
>>> Capabilities: [170 v1] Latency Tolerance Reporting
>>> Max snoop latency: 3145728ns
>>> Max no snoop latency: 3145728ns
>>> Capabilities: [178 v1] L1 PM Substates
>>> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
>>> PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
>>> L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
>>> T_CommonMode=0us LTR1.2_Threshold=0ns
>>> L1SubCtl2: T_PwrOn=10us
>>> Kernel driver in use: r8169
>>> Kernel modules: r8169
>>
>> Some Active State Power Management levels seem to be enabled.
>>
>>> Info from powertop about package C states. With ASPM my system reaches
>>> 50% PC7 + 50% PC10.
>>
>> That seems to be the case on my TUXEDO Book BU1406 too.
>>
>>> Paket | Kern | CPU 0 CPU 2
>>> | | C0 aktiv 1,7% 1,1%
>>> | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
>>> | | C1E 0,2% 0,8 ms 0,1% 0,2 ms
>>> C2 (pc2) 5,2% | |
>>> C3 (pc3) 82,1% | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,1% 0,2 ms
>
> Relevant are the package states and your system reaches pc3 only. The "Tunables" section
> in powertop may provide hints how to save more power.
Thank you for the hint. As it’s unrelated, I’ll just paste the tunables
below, but will try to forward it to the correct people.
Schlecht Audiocodec-Energieverwaltung einschalten
Schlecht VM-Rückschreibezeitlimit
>>> C6 (pc6) 0,0% | C6 (cc6) 1,3% | C6 0,8% 0,5 ms 1,4% 0,6 ms
>>> C7 (pc7) 0,0% | C7 (cc7) 90,8% | C7s 0,0% 1,6 ms 0,0% 0,0 ms
>>> C8 (pc8) 0,0% | | C8 6,0% 1,8 ms 10,1% 2,0 ms
>>> C9 (pc9) 0,0% | | C9 0,2% 2,8 ms 0,2% 2,9 ms
>>> C10 (pc10) 0,0% | | C10 88,7% 12,7 ms 84,4% 14,9 ms
>>>
>>> | Kern | CPU 1 CPU 3
>>> | | C0 aktiv 1,0% 0,8%
>>> | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
>>> | | C1E 0,1% 0,3 ms 0,1% 0,3 ms
>>> | |
>>> | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,0% 0,2 ms
>>> | C6 (cc6) 1,1% | C6 0,9% 0,6 ms 0,8% 0,5 ms
>>> | C7 (cc7) 92,2% | C7s 0,0% 1,7 ms 0,0% 0,0 ms
>>> | | C8 6,2% 1,7 ms 5,4% 1,7 ms
>>> | | C9 0,3% 1,7 ms 0,1% 1,9 ms
>>> | | C10 88,8% 12,1 ms 90,7% 14,8 ms
>>>
>>> | GPU |
>>> | |
>>> | Powered On 2,2% |
>>> | RC6 97,8% |
>>> | RC6p 0,0% |
>>> | RC6pp 0,0% |
>>
>>> dmesg output filtered for "r8169". Primarily relevant is the line with
>>> the chip name and XID.
>>
>> Please find them below.
>>
>>> $ sudo dmesg | grep r8169
>>> [ 5.318442] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 418
>>> [ 5.318470] r8169 0000:03:00.1: enabling device (0000 -> 0003)
>>> [ 5.340324] libphy: r8169: probed
>>> [ 5.340630] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 136
>
> Good to know. For this chip version rtl8168g_2_hw_phy_config() is used to configure the PHY,
> but this function just loads the firmware. So we don't know whether EEE is enabled.
>
> What you could do to test further is limiting the speed to 100MBit or 10MBit via ethtool.
> If this reduces power consumption significantly it's a hint that indeed the PHY seems
> to be the one to be blamed.
With `sudo ethtool -s enp3s0f1 speed 10 duplex full` the power usage
drops to 800 mW and even to 0, so it’s much less as with 1 Gbit/s.
>>> [ 5.340632] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
>>> [ 5.340673] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 9217 usecs
>>> [ 5.799967] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
>>> [ 10.036968] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
>>> [ 676.940934] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 22235
>>> [ 676.952411] libphy: r8169: probed
>>> [ 676.952701] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 139
>>> [ 676.952702] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
>>> [ 676.952736] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 11518 usecs
>>> [ 676.954420] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
>>> [ 676.975161] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
>>> [ 680.518923] r8169 0000:03:00.1 enp3s0f1: Link is Up - 1Gbps/Full - flow control rx/tx
>>> [ 1751.285899] r8169 0000:03:00.1: invalid short VPD tag 00 at offset 1
Kind regards,
Paul
^ permalink raw reply
* Re: [PATCH net] net: skb_scrub_packet(): Scrub offload_fwd_mark
From: Jiri Pirko @ 2018-11-20 11:41 UTC (permalink / raw)
To: Petr Machata; +Cc: netdev@vger.kernel.org, davem@davemloft.net, Ido Schimmel
In-Reply-To: <1bc608bd028b41a21c99c982f459b7434f0948ed.1542712365.git.petrm@mellanox.com>
Tue, Nov 20, 2018 at 12:39:56PM CET, petrm@mellanox.com wrote:
>When a packet is trapped and the corresponding SKB marked as
>already-forwarded, it retains this marking even after it is forwarded
>across veth links into another bridge. There, since it ingresses the
>bridge over veth, which doesn't have offload_fwd_mark, it triggers a
>warning in nbp_switchdev_frame_mark().
>
>Then nbp_switchdev_allowed_egress() decides not to allow egress from
>this bridge through another veth, because the SKB is already marked, and
>the mark (of 0) of course matches. Thus the packet is incorrectly
>blocked.
>
>Solve by resetting offload_fwd_mark() in skb_scrub_packet(). That
>function is called from tunnels and also from veth, and thus catches the
>cases where traffic is forwarded between bridges and transformed in a
>way that invalidates the marking.
>
>Fixes: 6bc506b4fb06 ("bridge: switchdev: Add forward mark support for stacked devices")
>Fixes: abf4bb6b63d0 ("skbuff: Add the offload_mr_fwd_mark field")
>Signed-off-by: Petr Machata <petrm@mellanox.com>
>Suggested-by: Ido Schimmel <idosch@mellanox.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
^ permalink raw reply
* [PATCH net] net: skb_scrub_packet(): Scrub offload_fwd_mark
From: Petr Machata @ 2018-11-20 11:39 UTC (permalink / raw)
To: netdev@vger.kernel.org; +Cc: davem@davemloft.net, Ido Schimmel
When a packet is trapped and the corresponding SKB marked as
already-forwarded, it retains this marking even after it is forwarded
across veth links into another bridge. There, since it ingresses the
bridge over veth, which doesn't have offload_fwd_mark, it triggers a
warning in nbp_switchdev_frame_mark().
Then nbp_switchdev_allowed_egress() decides not to allow egress from
this bridge through another veth, because the SKB is already marked, and
the mark (of 0) of course matches. Thus the packet is incorrectly
blocked.
Solve by resetting offload_fwd_mark() in skb_scrub_packet(). That
function is called from tunnels and also from veth, and thus catches the
cases where traffic is forwarded between bridges and transformed in a
way that invalidates the marking.
Fixes: 6bc506b4fb06 ("bridge: switchdev: Add forward mark support for stacked devices")
Fixes: abf4bb6b63d0 ("skbuff: Add the offload_mr_fwd_mark field")
Signed-off-by: Petr Machata <petrm@mellanox.com>
Suggested-by: Ido Schimmel <idosch@mellanox.com>
---
net/core/skbuff.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index b4ee5c8b928f..a8217e221e19 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -4854,6 +4854,11 @@ void skb_scrub_packet(struct sk_buff *skb, bool xnet)
nf_reset(skb);
nf_reset_trace(skb);
+#ifdef CONFIG_NET_SWITCHDEV
+ skb->offload_fwd_mark = 0;
+ skb->offload_mr_fwd_mark = 0;
+#endif
+
if (!xnet)
return;
--
2.4.11
^ permalink raw reply related
* Re: [PATCH 3/3] selftests: add a test for bpf_prog_test_run output size
From: Lorenz Bauer @ 2018-11-20 11:35 UTC (permalink / raw)
To: Y Song; +Cc: Alexei Starovoitov, Daniel Borkmann, netdev, linux-api
In-Reply-To: <CAH3MdRVd=ipYg+b9sGRR-FFUuOMjZ4OGZzkmJuFEShrYVXvwWA@mail.gmail.com>
On Sun, 18 Nov 2018 at 05:59, Y Song <ys114321@gmail.com> wrote:
>
> On Fri, Nov 16, 2018 at 12:55 PM Lorenz Bauer <lmb@cloudflare.com> wrote:
> >
> > Make sure that bpf_prog_test_run returns the correct length
> > in the size_out argument and that the kernel respects the
> > output size hint.
> >
> > Signed-off-by: Lorenz Bauer <lmb@cloudflare.com>
> > ---
> > tools/testing/selftests/bpf/test_progs.c | 34 ++++++++++++++++++++++++
> > 1 file changed, 34 insertions(+)
> >
> > diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c
> > index 560d7527b86b..6ab98e10e86f 100644
> > --- a/tools/testing/selftests/bpf/test_progs.c
> > +++ b/tools/testing/selftests/bpf/test_progs.c
> > @@ -124,6 +124,39 @@ static void test_pkt_access(void)
> > bpf_object__close(obj);
> > }
> >
> > +static void test_output_size_hint(void)
> > +{
> > + const char *file = "./test_pkt_access.o";
> > + struct bpf_object *obj;
> > + __u32 retval, size, duration;
> > + int err, prog_fd;
> > + char buf[10];
> > +
> > + err = bpf_prog_load(file, BPF_PROG_TYPE_SCHED_CLS, &obj, &prog_fd);
> > + if (err) {
> > + error_cnt++;
> > + return;
> > + }
> CHECK can also be used here.
> if (CHECK(...)) {
> goto done;
> }
> where label "done" is right before bpf_object__close.
I just copied this part from test_pkt_access, happy to change it though.
However, I think "goto done" would lead to freeing an unallocated
object in this case?
--
Lorenz Bauer | Systems Engineer
25 Lavington St., London SE1 0NZ
www.cloudflare.com
^ permalink raw reply
* Re: [PATCH v1 2/2] netfilter: Add PSID mode to MASQUERADE
From: kbuild test robot @ 2018-11-20 11:31 UTC (permalink / raw)
To: Felix Jia
Cc: kbuild-all, David S . Miller, Pablo Neira Ayuso, Blair Steven,
netdev, Anthony Lineham, Scott Parlane, Felix Jia
In-Reply-To: <20181120013114.22020-2-felix.jia@alliedtelesis.co.nz>
[-- Attachment #1: Type: text/plain, Size: 2042 bytes --]
Hi Blair,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on nf-next/master]
[also build test ERROR on v4.20-rc3 next-20181120]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Felix-Jia/ip6_tunnel-Adding-support-of-mapping-rules-for-MAP-E-tunnel/20181120-173346
base: https://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next.git master
config: sh-allyesconfig (attached as .config)
compiler: sh4-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.2.0 make.cross ARCH=sh
All errors (new ones prefixed by >>):
>> net/ipv6/netfilter/nf_nat_proto_icmpv6.c:85:15: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
.in_range = icmpv6_in_range,
^~~~~~~~~~~~~~~
net/ipv6/netfilter/nf_nat_proto_icmpv6.c:85:15: note: (near initialization for 'nf_nat_l4proto_icmpv6.in_range')
cc1: some warnings being treated as errors
vim +85 net/ipv6/netfilter/nf_nat_proto_icmpv6.c
58a317f1 Patrick McHardy 2012-08-26 81
58a317f1 Patrick McHardy 2012-08-26 82 const struct nf_nat_l4proto nf_nat_l4proto_icmpv6 = {
58a317f1 Patrick McHardy 2012-08-26 83 .l4proto = IPPROTO_ICMPV6,
58a317f1 Patrick McHardy 2012-08-26 84 .manip_pkt = icmpv6_manip_pkt,
58a317f1 Patrick McHardy 2012-08-26 @85 .in_range = icmpv6_in_range,
:::::: The code at line 85 was first introduced by commit
:::::: 58a317f1061c894d2344c0b6a18ab4a64b69b815 netfilter: ipv6: add IPv6 NAT support
:::::: TO: Patrick McHardy <kaber@trash.net>
:::::: CC: Pablo Neira Ayuso <pablo@netfilter.org>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 50565 bytes --]
^ permalink raw reply
* Re: [RFC PATCH] net: don't keep lonely packets forever in the gro hash
From: Sergei Shtylyov @ 2018-11-20 11:30 UTC (permalink / raw)
To: Paolo Abeni, netdev; +Cc: David S. Miller, Willem de Bruijn, Eric Dumazet
In-Reply-To: <3c8b5aea0c812323d8e15b548789a1e240f499d7.1542709015.git.pabeni@redhat.com>
On 11/20/2018 01:17 PM, Paolo Abeni wrote:
> Eric noted that with UDP GRO and napi timeout, we could keep a single
NAPI, else you're simply inconsistent.
> UDP packet inside the GRO hash forever, if the related NAPI instance
> calls napi_gro_complete() at an higher frequency than the napi timeout.
NAPI.
> Willem noted that even TCP packets could be trapped there, till the
> next retransmission.
> This patch tries to address the issue, flushing the oldest packets before
> scheduling the NAPI timeout. The rationale is that such a timeout should be
> well below a jiffy and we are not flushing packets eligible for sane GRO.
>
> Reported-by: Eric Dumazet <eric.dumazet@gmail.com>
> Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> ---
> Sending as RFC, as I fear I'm missing some relevant pieces.
> Also I'm unsure if this should considered a fixes for "udp: implement
> GRO for plain UDP sockets." or for "net: gro: add a per device gro flush timer"
> ---
> net/core/dev.c | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 5927f6a7c301..5cc4c4961869 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -5975,11 +5975,14 @@ bool napi_complete_done(struct napi_struct *n, int work_done)
> if (work_done)
> timeout = n->dev->gro_flush_timeout;
>
> + /* When the NAPI instance uses a timeout, we still need to
> + * someout bound the time packets are keept in the GRO layer
Somehow?
> + * under heavy traffic
> + */
> + napi_gro_flush(n, !!timeout);
> if (timeout)
> hrtimer_start(&n->timer, ns_to_ktime(timeout),
> HRTIMER_MODE_REL_PINNED);
> - else
> - napi_gro_flush(n, false);
> }
> if (unlikely(!list_empty(&n->poll_list))) {
> /* If n->poll_list is not empty, we need to mask irqs */
MBR, Sergei
^ permalink raw reply
* Re: [PATCH bpf-next 1/2] bpf: adding support for map in map in libbpf
From: Edward Cree @ 2018-11-20 11:29 UTC (permalink / raw)
To: Nikita V. Shirokov, Alexei Starovoitov, Daniel Borkmann,
Jakub Kicinski
Cc: netdev
In-Reply-To: <20181120000626.16825-2-tehnerd@tehnerd.com>
On 20/11/18 00:06, Nikita V. Shirokov wrote:
> idea is pretty simple. for specified map (pointed by struct bpf_map)
> we would provide descriptor of already loaded map, which is going to be
> used as a prototype for inner map. proposed workflow:
> 1) open bpf's object (bpf_object__open)
> 2) create bpf's map which is going to be used as a prototype
> 3) find (by name) map-in-map which you want to load and update w/
> descriptor of inner map w/ a new helper from this patch
> 4) load bpf program w/ bpf_object__load
>
> inner_map_fd is ignored by any other maps asidef from (hash|array) of
"asidef" -> "aside"
^ permalink raw reply
* Re: [PATCH v5 bpf-next 00/13] bpf: Add btf func info support
From: Edward Cree @ 2018-11-20 11:26 UTC (permalink / raw)
To: Martin KaFai Lau, netdev; +Cc: Alexei Starovoitov, Daniel Borkmann, kernel-team
In-Reply-To: <20181119232906.144809-1-kafai@fb.com>
On 19/11/18 23:29, Martin KaFai Lau wrote:
> The BTF support was added to kernel by Commit 69b693f0aefa
> ("bpf: btf: Introduce BPF Type Format (BTF)"), which introduced
> .BTF section into ELF file and is primarily
> used for map pretty print.
> pahole is used to convert dwarf to BTF for ELF files.
>
> This patch added func info support to the kernel so we can
> get better ksym's for bpf function calls. Basically,
> function call types are passed to kernel and the kernel
> extract function names from these types in order to contruct ksym
> for these functions.
>
> [...]
Acked-by: Edward Cree <ecree@solarflare.com>
^ permalink raw reply
* [PATCH net] sctp: hold transport before accessing its asoc in sctp_epaddr_lookup_transport
From: Xin Long @ 2018-11-20 11:12 UTC (permalink / raw)
To: network dev, linux-sctp; +Cc: davem, Marcelo Ricardo Leitner, Neil Horman
Without holding transport to dereference its asoc, a use after
free panic can be caused in sctp_epaddr_lookup_transport. Note
that a sock lock can't protect these transports that belong to
other socks.
A similar fix as Commit bab1be79a516 ("sctp: hold transport
before accessing its asoc in sctp_transport_get_next") is
needed to hold the transport before accessing its asoc in
sctp_epaddr_lookup_transport.
Fixes: 7fda702f9315 ("sctp: use new rhlist interface on sctp transport rhashtable")
Reported-by: syzbot+aad231d51b1923158444@syzkaller.appspotmail.com
Signed-off-by: Xin Long <lucien.xin@gmail.com>
---
net/sctp/input.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/net/sctp/input.c b/net/sctp/input.c
index 69584e9..c2c0816 100644
--- a/net/sctp/input.c
+++ b/net/sctp/input.c
@@ -972,9 +972,15 @@ struct sctp_transport *sctp_epaddr_lookup_transport(
list = rhltable_lookup(&sctp_transport_hashtable, &arg,
sctp_hash_params);
- rhl_for_each_entry_rcu(t, tmp, list, node)
- if (ep == t->asoc->ep)
+ rhl_for_each_entry_rcu(t, tmp, list, node) {
+ if (!sctp_transport_hold(t))
+ continue;
+ if (ep == t->asoc->ep) {
+ sctp_transport_put(t);
return t;
+ }
+ sctp_transport_put(t);
+ }
return NULL;
}
--
2.1.0
^ permalink raw reply related
* [PATCH net] sctp: hold transport before accessing its asoc in sctp_hash_transport
From: Xin Long @ 2018-11-20 11:09 UTC (permalink / raw)
To: network dev, linux-sctp; +Cc: davem, Marcelo Ricardo Leitner, Neil Horman
In sctp_hash_transport, it dereferences a transport's asoc only under
rcu_read_lock. Without holding the transport, its asoc could be freed
already, which leads to a use-after-free panic.
A similar fix as Commit bab1be79a516 ("sctp: hold transport before
accessing its asoc in sctp_transport_get_next") is needed to hold
the transport before accessing its asoc in sctp_hash_transport.
Fixes: cd2b70875058 ("sctp: check duplicate node before inserting a new transport")
Reported-by: syzbot+0b05d8aa7cb185107483@syzkaller.appspotmail.com
Signed-off-by: Xin Long <lucien.xin@gmail.com>
---
net/sctp/input.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/net/sctp/input.c b/net/sctp/input.c
index 5c36a99..69584e9 100644
--- a/net/sctp/input.c
+++ b/net/sctp/input.c
@@ -896,11 +896,16 @@ int sctp_hash_transport(struct sctp_transport *t)
list = rhltable_lookup(&sctp_transport_hashtable, &arg,
sctp_hash_params);
- rhl_for_each_entry_rcu(transport, tmp, list, node)
+ rhl_for_each_entry_rcu(transport, tmp, list, node) {
+ if (!sctp_transport_hold(transport))
+ continue;
if (transport->asoc->ep == t->asoc->ep) {
+ sctp_transport_put(transport);
rcu_read_unlock();
return -EEXIST;
}
+ sctp_transport_put(transport);
+ }
rcu_read_unlock();
err = rhltable_insert_key(&sctp_transport_hashtable, &arg,
--
2.1.0
^ permalink raw reply related
* Re: Realtek NIC uses over 1 Watt with no traffic
From: Heiner Kallweit @ 2018-11-20 21:06 UTC (permalink / raw)
To: Paul Menzel, Andrew Lunn
Cc: Realtek linux nic maintainers, David S. Miller, netdev,
linux-kernel
In-Reply-To: <524e54fa-c7b2-7cf5-7bd1-7b9ad7a64a46@molgen.mpg.de>
On 20.11.2018 21:31, Paul Menzel wrote:
> Dear Heiner,
>
>
> Thank you for your reply.
>
> Am 20.11.18 um 21:14 schrieb Heiner Kallweit:
>> On 20.11.2018 15:45, Andrew Lunn wrote:
>>> On Tue, Nov 20, 2018 at 09:40:25AM +0100, Paul Menzel wrote:
>
>>>> Using Ubuntu 18.10, Linux 4.18.0-11-generic, PowerTOP 2.9 shows, the NIC
>>>> uses 1.77 Watts. A network cable is plugged in, but there is no real traffic
>>>> according to `iftop`. Only an email program is running.
>>>>
>>>> $ lspci -nn -s 3:00.1
>>>> 03:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
>>>> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev
>>>> 12)
>>>>
>>>> Is that a measurement error, or does the NIC really need that much power?
>
>>> This sounds like Energy Efficient Ethernet, EEE, is not enabled.
>>>
>>> What does ethtool --show-eee ethX say?
>
> $ sudo ethtool --show-eee enp3s0f1
> Cannot get EEE settings: Operation not supported
>
>> The r8169 driver doesn't support the get_eee ethtool_ops callback.
>> For certain chip versions EEE gets enabled in the PHY init, for others
>> not and some don't seem to support EEE at all.
>>
>> Apart from EEE one important factor affecting power consumption is ASPM.
>> This was recently enabled for certain chip versions.
>>
>> Information that would help:
>>
>> whether Wake-on-LAN is enabled ("Wake-on:" line from ethtool output)
>
> ```
> $ sudo ethtool enp3s0f1
> Settings for enp3s0f1:
> Supported ports: [ TP AUI BNC MII FIBRE ]
> Supported link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Supported pause frame use: Symmetric Receive-only
> Supports auto-negotiation: Yes
> Supported FEC modes: Not reported
> Advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Advertised pause frame use: Symmetric Receive-only
> Advertised auto-negotiation: Yes
> Advertised FEC modes: Not reported
> Link partner advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Link partner advertised pause frame use: Symmetric
> Link partner advertised auto-negotiation: Yes
> Link partner advertised FEC modes: Not reported
> Speed: 1000Mb/s
> Duplex: Full
> Port: MII
> PHYAD: 0
> Transceiver: internal
> Auto-negotiation: on
> Supports Wake-on: pumbg
> Wake-on: g
> Current message level: 0x00000033 (51)
> drv probe ifdown ifup
> Link detected: yes
> ```
>
> So, it’s enabled (g Wake on MagicPacket(tm)).
>
> Running `sudo ethtool -s enp3s0f1 wol d;` doesn’t change anything though.
>
>> lspci -vv output for the Realtek NIC
>
> Here is the output (quoted, so that Thunderbird does not wrap the line).
>
>> $ sudo lspci -vv -s 3:00.1
>> 03:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 12)
>> Subsystem: CLEVO/KAPOK Computer RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> Latency: 0, Cache Line Size: 64 bytes
>> Interrupt: pin A routed to IRQ 19
>> Region 0: I/O ports at e000 [size=256]
>> Region 2: Memory at df114000 (64-bit, non-prefetchable) [size=4K]
>> Region 4: Memory at df110000 (64-bit, non-prefetchable) [size=16K]
>> Capabilities: [40] Power Management version 3
>> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>> Address: 0000000000000000 Data: 0000
>> Capabilities: [70] Express (v2) Endpoint, MSI 01
>> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
>> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
>> MaxPayload 128 bytes, MaxReadReq 4096 bytes
>> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
>> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
>> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>> LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
L0s is missing here, no idea why.
>> ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
>> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE#
>> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
>> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
>> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>> Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
>> Vector table: BAR=4 offset=00000000
>> PBA: BAR=4 offset=00000800
>> Capabilities: [d0] Vital Product Data
>> pcilib: sysfs_read_vpd: read failed: Input/output error
>> Not readable
>> Capabilities: [100 v2] Advanced Error Reporting
>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>> CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ NonFatalErr+
>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>> Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
>> Capabilities: [170 v1] Latency Tolerance Reporting
>> Max snoop latency: 3145728ns
>> Max no snoop latency: 3145728ns
>> Capabilities: [178 v1] L1 PM Substates
>> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
>> PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
>> L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
>> T_CommonMode=0us LTR1.2_Threshold=0ns
>> L1SubCtl2: T_PwrOn=10us
>> Kernel driver in use: r8169
>> Kernel modules: r8169
>
> Some Active State Power Management levels seem to be enabled.
>
>> Info from powertop about package C states. With ASPM my system reaches
>> 50% PC7 + 50% PC10.
>
> That seems to be the case on my TUXEDO Book BU1406 too.
>
>> Paket | Kern | CPU 0 CPU 2
>> | | C0 aktiv 1,7% 1,1%
>> | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
>> | | C1E 0,2% 0,8 ms 0,1% 0,2 ms
>> C2 (pc2) 5,2% | |
>> C3 (pc3) 82,1% | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,1% 0,2 ms
Relevant are the package states and your system reaches pc3 only. The "Tunables" section
in powertop may provide hints how to save more power.
>> C6 (pc6) 0,0% | C6 (cc6) 1,3% | C6 0,8% 0,5 ms 1,4% 0,6 ms
>> C7 (pc7) 0,0% | C7 (cc7) 90,8% | C7s 0,0% 1,6 ms 0,0% 0,0 ms
>> C8 (pc8) 0,0% | | C8 6,0% 1,8 ms 10,1% 2,0 ms
>> C9 (pc9) 0,0% | | C9 0,2% 2,8 ms 0,2% 2,9 ms
>> C10 (pc10) 0,0% | | C10 88,7% 12,7 ms 84,4% 14,9 ms
>>
>> | Kern | CPU 1 CPU 3
>> | | C0 aktiv 1,0% 0,8%
>> | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
>> | | C1E 0,1% 0,3 ms 0,1% 0,3 ms
>> | |
>> | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,0% 0,2 ms
>> | C6 (cc6) 1,1% | C6 0,9% 0,6 ms 0,8% 0,5 ms
>> | C7 (cc7) 92,2% | C7s 0,0% 1,7 ms 0,0% 0,0 ms
>> | | C8 6,2% 1,7 ms 5,4% 1,7 ms
>> | | C9 0,3% 1,7 ms 0,1% 1,9 ms
>> | | C10 88,8% 12,1 ms 90,7% 14,8 ms
>>
>> | GPU |
>> | |
>> | Powered On 2,2% |
>> | RC6 97,8% |
>> | RC6p 0,0% |
>> | RC6pp 0,0% |
>
>> dmesg output filtered for "r8169". Primarily relevant is the line with
>> the chip name and XID.
>
> Please find them below.
>
>> $ sudo dmesg | grep r8169
>> [ 5.318442] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 418
>> [ 5.318470] r8169 0000:03:00.1: enabling device (0000 -> 0003)
>> [ 5.340324] libphy: r8169: probed
>> [ 5.340630] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 136
Good to know. For this chip version rtl8168g_2_hw_phy_config() is used to configure the PHY,
but this function just loads the firmware. So we don't know whether EEE is enabled.
What you could do to test further is limiting the speed to 100MBit or 10MBit via ethtool.
If this reduces power consumption significantly it's a hint that indeed the PHY seems
to be the one to be blamed.
>> [ 5.340632] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
>> [ 5.340673] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 9217 usecs
>> [ 5.799967] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
>> [ 10.036968] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
>> [ 676.940934] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 22235
>> [ 676.952411] libphy: r8169: probed
>> [ 676.952701] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 139
>> [ 676.952702] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
>> [ 676.952736] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 11518 usecs
>> [ 676.954420] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
>> [ 676.975161] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
>> [ 680.518923] r8169 0000:03:00.1 enp3s0f1: Link is Up - 1Gbps/Full - flow control rx/tx
>> [ 1751.285899] r8169 0000:03:00.1: invalid short VPD tag 00 at offset 1
>
>
> Kind regards,
>
> Paul
>
Heiner
^ permalink raw reply
* [RFC PATCH] net: don't keep lonely packets forever in the gro hash
From: Paolo Abeni @ 2018-11-20 10:17 UTC (permalink / raw)
To: netdev; +Cc: David S. Miller, Willem de Bruijn, Eric Dumazet
Eric noted that with UDP GRO and napi timeout, we could keep a single
UDP packet inside the GRO hash forever, if the related NAPI instance
calls napi_gro_complete() at an higher frequency than the napi timeout.
Willem noted that even TCP packets could be trapped there, till the
next retransmission.
This patch tries to address the issue, flushing the oldest packets before
scheduling the NAPI timeout. The rationale is that such a timeout should be
well below a jiffy and we are not flushing packets eligible for sane GRO.
Reported-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
---
Sending as RFC, as I fear I'm missing some relevant pieces.
Also I'm unsure if this should considered a fixes for "udp: implement
GRO for plain UDP sockets." or for "net: gro: add a per device gro flush timer"
---
net/core/dev.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/net/core/dev.c b/net/core/dev.c
index 5927f6a7c301..5cc4c4961869 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -5975,11 +5975,14 @@ bool napi_complete_done(struct napi_struct *n, int work_done)
if (work_done)
timeout = n->dev->gro_flush_timeout;
+ /* When the NAPI instance uses a timeout, we still need to
+ * someout bound the time packets are keept in the GRO layer
+ * under heavy traffic
+ */
+ napi_gro_flush(n, !!timeout);
if (timeout)
hrtimer_start(&n->timer, ns_to_ktime(timeout),
HRTIMER_MODE_REL_PINNED);
- else
- napi_gro_flush(n, false);
}
if (unlikely(!list_empty(&n->poll_list))) {
/* If n->poll_list is not empty, we need to mask irqs */
--
2.17.2
^ permalink raw reply related
* Re: Realtek NIC uses over 1 Watt with no traffic
From: Andrew Lunn @ 2018-11-20 20:37 UTC (permalink / raw)
To: Paul Menzel
Cc: Realtek linux nic maintainers, David S. Miller, netdev,
linux-kernel, Heiner Kallweit
In-Reply-To: <7714a7b7-ca22-19ea-499a-e2d60f3c9fb5@molgen.mpg.de>
> >This sounds like Energy Efficient Ethernet, EEE, is not enabled.
>
> No, idea if the router from my provider supports that.
It is pretty standard these days. But it requires both local and
remove MAC and PHY support it.
> >What does ethtool --show-eee ethX say?
>
> $ sudo ethtool --show-eee enp3s0f1
> Cannot get EEE settings: Operation not supported
It would be nice to support this. Maybe Heiner can put it onto his
TODO list.
Andrew
^ permalink raw reply
* Re: Realtek NIC uses over 1 Watt with no traffic
From: Paul Menzel @ 2018-11-20 20:31 UTC (permalink / raw)
To: Heiner Kallweit, Andrew Lunn
Cc: Realtek linux nic maintainers, David S. Miller, netdev,
linux-kernel
In-Reply-To: <c3ff00a1-ac51-7f27-8e90-b753f316866d@gmail.com>
Dear Heiner,
Thank you for your reply.
Am 20.11.18 um 21:14 schrieb Heiner Kallweit:
> On 20.11.2018 15:45, Andrew Lunn wrote:
>> On Tue, Nov 20, 2018 at 09:40:25AM +0100, Paul Menzel wrote:
>>> Using Ubuntu 18.10, Linux 4.18.0-11-generic, PowerTOP 2.9 shows, the NIC
>>> uses 1.77 Watts. A network cable is plugged in, but there is no real traffic
>>> according to `iftop`. Only an email program is running.
>>>
>>> $ lspci -nn -s 3:00.1
>>> 03:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
>>> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev
>>> 12)
>>>
>>> Is that a measurement error, or does the NIC really need that much power?
>> This sounds like Energy Efficient Ethernet, EEE, is not enabled.
>>
>> What does ethtool --show-eee ethX say?
$ sudo ethtool --show-eee enp3s0f1
Cannot get EEE settings: Operation not supported
> The r8169 driver doesn't support the get_eee ethtool_ops callback.
> For certain chip versions EEE gets enabled in the PHY init, for others
> not and some don't seem to support EEE at all.
>
> Apart from EEE one important factor affecting power consumption is ASPM.
> This was recently enabled for certain chip versions.
>
> Information that would help:
>
> whether Wake-on-LAN is enabled ("Wake-on:" line from ethtool output)
```
$ sudo ethtool enp3s0f1
Settings for enp3s0f1:
Supported ports: [ TP AUI BNC MII FIBRE ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000033 (51)
drv probe ifdown ifup
Link detected: yes
```
So, it’s enabled (g Wake on MagicPacket(tm)).
Running `sudo ethtool -s enp3s0f1 wol d;` doesn’t change anything though.
> lspci -vv output for the Realtek NIC
Here is the output (quoted, so that Thunderbird does not wrap the line).
> $ sudo lspci -vv -s 3:00.1
> 03:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 12)
> Subsystem: CLEVO/KAPOK Computer RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin A routed to IRQ 19
> Region 0: I/O ports at e000 [size=256]
> Region 2: Memory at df114000 (64-bit, non-prefetchable) [size=4K]
> Region 4: Memory at df110000 (64-bit, non-prefetchable) [size=16K]
> Capabilities: [40] Power Management version 3
> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
> Address: 0000000000000000 Data: 0000
> Capabilities: [70] Express (v2) Endpoint, MSI 01
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
> MaxPayload 128 bytes, MaxReadReq 4096 bytes
> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
> LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
> ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE#
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
> Vector table: BAR=4 offset=00000000
> PBA: BAR=4 offset=00000800
> Capabilities: [d0] Vital Product Data
> pcilib: sysfs_read_vpd: read failed: Input/output error
> Not readable
> Capabilities: [100 v2] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ NonFatalErr+
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
> Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
> Capabilities: [170 v1] Latency Tolerance Reporting
> Max snoop latency: 3145728ns
> Max no snoop latency: 3145728ns
> Capabilities: [178 v1] L1 PM Substates
> L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
> PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
> L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
> T_CommonMode=0us LTR1.2_Threshold=0ns
> L1SubCtl2: T_PwrOn=10us
> Kernel driver in use: r8169
> Kernel modules: r8169
Some Active State Power Management levels seem to be enabled.
> Info from powertop about package C states. With ASPM my system reaches
> 50% PC7 + 50% PC10.
That seems to be the case on my TUXEDO Book BU1406 too.
> Paket | Kern | CPU 0 CPU 2
> | | C0 aktiv 1,7% 1,1%
> | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
> | | C1E 0,2% 0,8 ms 0,1% 0,2 ms
> C2 (pc2) 5,2% | |
> C3 (pc3) 82,1% | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,1% 0,2 ms
> C6 (pc6) 0,0% | C6 (cc6) 1,3% | C6 0,8% 0,5 ms 1,4% 0,6 ms
> C7 (pc7) 0,0% | C7 (cc7) 90,8% | C7s 0,0% 1,6 ms 0,0% 0,0 ms
> C8 (pc8) 0,0% | | C8 6,0% 1,8 ms 10,1% 2,0 ms
> C9 (pc9) 0,0% | | C9 0,2% 2,8 ms 0,2% 2,9 ms
> C10 (pc10) 0,0% | | C10 88,7% 12,7 ms 84,4% 14,9 ms
>
> | Kern | CPU 1 CPU 3
> | | C0 aktiv 1,0% 0,8%
> | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
> | | C1E 0,1% 0,3 ms 0,1% 0,3 ms
> | |
> | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,0% 0,2 ms
> | C6 (cc6) 1,1% | C6 0,9% 0,6 ms 0,8% 0,5 ms
> | C7 (cc7) 92,2% | C7s 0,0% 1,7 ms 0,0% 0,0 ms
> | | C8 6,2% 1,7 ms 5,4% 1,7 ms
> | | C9 0,3% 1,7 ms 0,1% 1,9 ms
> | | C10 88,8% 12,1 ms 90,7% 14,8 ms
>
> | GPU |
> | |
> | Powered On 2,2% |
> | RC6 97,8% |
> | RC6p 0,0% |
> | RC6pp 0,0% |
> dmesg output filtered for "r8169". Primarily relevant is the line with
> the chip name and XID.
Please find them below.
> $ sudo dmesg | grep r8169
> [ 5.318442] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 418
> [ 5.318470] r8169 0000:03:00.1: enabling device (0000 -> 0003)
> [ 5.340324] libphy: r8169: probed
> [ 5.340630] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 136
> [ 5.340632] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
> [ 5.340673] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 9217 usecs
> [ 5.799967] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
> [ 10.036968] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
> [ 676.940934] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 22235
> [ 676.952411] libphy: r8169: probed
> [ 676.952701] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 139
> [ 676.952702] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
> [ 676.952736] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 11518 usecs
> [ 676.954420] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
> [ 676.975161] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
> [ 680.518923] r8169 0000:03:00.1 enp3s0f1: Link is Up - 1Gbps/Full - flow control rx/tx
> [ 1751.285899] r8169 0000:03:00.1: invalid short VPD tag 00 at offset 1
Kind regards,
Paul
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox