* Re: [Intel-wired-lan] [PATCH 1/2] e1000e: Don't return uninitialized stats
From: Benjamin Poirier @ 2017-04-24 19:01 UTC (permalink / raw)
To: Paul Menzel; +Cc: Jeff Kirsher, netdev, intel-wired-lan, Stefan Priebe
In-Reply-To: <71d8032d-5ed1-8242-83e6-16ce65718966@molgen.mpg.de>
On 2017/04/24 10:23, Paul Menzel wrote:
> Dear Benjamin,
>
>
> Thank you for your fix.
>
> On 04/21/17 23:20, Benjamin Poirier wrote:
> > Some statistics passed to ethtool are garbage because e1000e_get_stats64()
> > doesn't write them, for example: tx_heartbeat_errors. This leaks kernel
> > memory to userspace and confuses users.
>
> Could you please give specific examples to reproduce the issue? That way
> your fix can also be tested.
>
Some fields in e1000_get_ethtool_stats()'s net_stats are not initialized
by e1000e_get_stats64(). The structure is allocated on the stack,
therefore, the value of those fields depends on previous stack content;
that in turns depends on kernel version, compiler and previous execution
path. I've tried on 8 machines with different kernel versions and it
reproduced on 3.
root@linux-zxe0:/usr/local/src/linux# git log -n1 --oneline
fc1f8f4f310a net: ipv6: send unsolicited NA if enabled for all interfaces
root@linux-zxe0:/usr/local/src/linux# ethtool -i eth0
driver: e1000e
[...]
root@linux-zxe0:/usr/local/src/linux# ethtool -S eth0
NIC statistics:
rx_packets: 217
tx_packets: 153
rx_bytes: 23091
tx_bytes: 20533
rx_broadcast: 0
tx_broadcast: 6
rx_multicast: 0
tx_multicast: 10
rx_errors: 0
tx_errors: 0
tx_dropped: 18446683600612146192
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 70364470214850
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 18446744072101618112
tx_heartbeat_errors: 18446612150964469760
[...]
(gdb) p /x 18446683600612146192
$1 = 0xffffc9000282bc10
(gdb) p /x 18446744072101618112
$2 = 0xffffffffa028e1c0
(gdb) p /x 18446612150964469760
$3 = 0xffff880457a44000
... a bunch of kernel addresses
Inserting a dummy memset is a reliable way to show the issue:
--- a/drivers/net/ethernet/intel/e1000e/ethtool.c
+++ b/drivers/net/ethernet/intel/e1000e/ethtool.c
@@ -2061,6 +2061,8 @@ static void e1000_get_ethtool_stats(struct net_device *netdev,
int i;
char *p = NULL;
+ memset(&net_stats, 0xff, sizeof(net_stats));
+
pm_runtime_get_sync(netdev->dev.parent);
e1000e_get_stats64(netdev, &net_stats);
root@linux-zxe0:/usr/local/src/linux# ethtool -S eth0
NIC statistics:
rx_packets: 30
tx_packets: 29
rx_bytes: 2924
tx_bytes: 3012
rx_broadcast: 0
tx_broadcast: 6
rx_multicast: 0
tx_multicast: 7
rx_errors: 0
tx_errors: 0
tx_dropped: 18446744073709551615
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 18446744073709551615
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 18446744073709551615
tx_heartbeat_errors: 18446744073709551615
[...]
(gdb) p /x 18446744073709551615
$1 = 0xffffffffffffffff
^ permalink raw reply
* Re: [Intel-wired-lan] [PATCH 1/2] e1000e: Don't return uninitialized stats
From: Benjamin Poirier @ 2017-04-24 19:10 UTC (permalink / raw)
To: Neftin, Sasha
Cc: netdev@vger.kernel.org, intel-wired-lan, Ruinskiy, Dima, Kirsher,
Jeffrey T, Stefan Priebe
In-Reply-To: <caef8d69-9e47-bbcc-0758-e5900e1cb291@intel.com>
Sasha, please use reply-all to keep everyone in cc (including me...).
On 2017/04/24 11:17, Neftin, Sasha wrote:
> On 4/23/2017 15:53, Neftin, Sasha wrote:
> > -----Original Message-----
> > From: Intel-wired-lan [mailto:intel-wired-lan-bounces@lists.osuosl.org] On Behalf Of Benjamin Poirier
> > Sent: Saturday, April 22, 2017 00:20
> > To: Kirsher, Jeffrey T <jeffrey.t.kirsher@intel.com>
> > Cc: netdev@vger.kernel.org; intel-wired-lan@lists.osuosl.org; Stefan Priebe <s.priebe@profihost.ag>
> > Subject: [Intel-wired-lan] [PATCH 1/2] e1000e: Don't return uninitialized stats
> >
> > Some statistics passed to ethtool are garbage because e1000e_get_stats64() doesn't write them, for example: tx_heartbeat_errors. This leaks kernel memory to userspace and confuses users.
> >
> > Do like ixgbe and use dev_get_stats() which first zeroes out rtnl_link_stats64.
> >
> > Reported-by: Stefan Priebe <s.priebe@profihost.ag>
> > Signed-off-by: Benjamin Poirier <bpoirier@suse.com>
> > ---
> > drivers/net/ethernet/intel/e1000e/ethtool.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/net/ethernet/intel/e1000e/ethtool.c b/drivers/net/ethernet/intel/e1000e/ethtool.c
> > index 7aff68a4a4df..f117b90cdc2f 100644
> > --- a/drivers/net/ethernet/intel/e1000e/ethtool.c
> > +++ b/drivers/net/ethernet/intel/e1000e/ethtool.c
> > @@ -2063,7 +2063,7 @@ static void e1000_get_ethtool_stats(struct net_device *netdev,
> > pm_runtime_get_sync(netdev->dev.parent);
> > - e1000e_get_stats64(netdev, &net_stats);
> > + dev_get_stats(netdev, &net_stats);
> > pm_runtime_put_sync(netdev->dev.parent);
> > --
> > 2.12.2
> >
> > _______________________________________________
> > Intel-wired-lan mailing list
> > Intel-wired-lan@lists.osuosl.org
> > http://lists.osuosl.org/mailman/listinfo/intel-wired-lan
>
> Hello,
>
> We would like to not accept this patch. Suggested generic method
> '*dev_get_stats' (net/core/dev.c) calls 'ops->ndo_get_stats64' method which
> eventually calls e1000e_get_stats64 (netdev.c) - so there is same
> functionality. Also, see that 'e1000e_get_stats64' method in netdev.c (line
No, it's not the same functionality because dev_get_stats() does a
memset on the rtnl_link_stats64 struct.
> 5928) calls 'memset' with 0's before update statistics. Local sanity check
I don't see any memset in e1000e_get_stats64(). What kernel version are
you looking at?
> in our lab shows 'tx_heartbeat_errors' counter reported as 0.
>
Please see the mail I just sent to Paul Menzel <pmenzel@molgen.mpg.de>
for more information about the issue and how to reproduce it.
^ permalink raw reply
* Re: [PATCH v2] brcmfmac: Make skb header writable before use
From: Arend Van Spriel @ 2017-04-24 19:14 UTC (permalink / raw)
To: James Hughes, Franky Lin, Hante Meuleman, Kalle Valo,
linux-wireless, brcm80211-dev-list.pdl, netdev
In-Reply-To: <20170424130322.476-1-james.hughes@raspberrypi.org>
On 24-4-2017 15:03, James Hughes wrote:
> The driver was making changes to the skb_header without
> ensuring it was writable (i.e. uncloned).
> This patch also removes some boiler plate header size
> checking/adjustment code as that is also handled by the
> skb_cow_header function used to make header writable.
>
> This patch depends on
> brcmfmac: Ensure pointer correctly set if skb data location changes
Hi James,
I actually thought I was going to tackle this so I spend some time
working on it today, but I will submit that as a subsequent patch. Below
some comments but apart from that:
Acked-by: Arend van Spriel <arend.vanspriel@broadcom.com>
> Signed-off-by: James Hughes <james.hughes@raspberrypi.org>
> ---
> Changes in v2
> Makes the _cow_ call at the entry point of the skb in to the
> stack, means only needs to be done once, and error handling
> is easier.
> Split a separate minor bug fix off to a separate patch (which
> this patch depends on)
>
>
>
> .../net/wireless/broadcom/brcm80211/brcmfmac/core.c | 19 ++++---------------
> 1 file changed, 4 insertions(+), 15 deletions(-)
>
> diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
> index 9b7c19a508ac..88f8675a94c2 100644
> --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
> +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
> @@ -211,22 +211,11 @@ static netdev_tx_t brcmf_netdev_start_xmit(struct sk_buff *skb,
> goto done;
> }
>
> - /* Make sure there's enough room for any header */
> - if (skb_headroom(skb) < drvr->hdrlen) {
> - struct sk_buff *skb2;
> -
> - brcmf_dbg(INFO, "%s: insufficient headroom\n",
> - brcmf_ifname(ifp));
> - drvr->bus_if->tx_realloc++;
> - skb2 = skb_realloc_headroom(skb, drvr->hdrlen);
> + /* Make sure there's enough room for any header, and make it writable */
Please rephrase: /* Make sure there is enough writable headroom */
Just do:
ret = skb_cow_head(skb, drvr->hdrlen);
if (ret < 0) {
brcmf_err("%s: skb_cow_head failed\n",
brcmf_ifname(ifp));
> + if (skb_cow_head(skb, drvr->hdrlen)) {
> dev_kfree_skb(skb);
> - skb = skb2;
> - if (skb == NULL) {
> - brcmf_err("%s: skb_realloc_headroom failed\n",
> - brcmf_ifname(ifp));
> - ret = -ENOMEM;
> - goto done;
> - }
> + ret = -ENOMEM;
... and drop this assignment.
> + goto done;
> }
>
> /* validate length for ether packet */
>
^ permalink raw reply
* Re: [Intel-wired-lan] [PATCH 1/2] e1000e: Don't return uninitialized stats
From: David Miller @ 2017-04-24 19:15 UTC (permalink / raw)
To: bpoirier; +Cc: pmenzel, jeffrey.t.kirsher, netdev, intel-wired-lan, s.priebe
In-Reply-To: <20170424190151.cbmbbeyjspoolpho@f1.synalogic.ca>
From: Benjamin Poirier <bpoirier@suse.com>
Date: Mon, 24 Apr 2017 12:01:51 -0700
> On 2017/04/24 10:23, Paul Menzel wrote:
>> Dear Benjamin,
>>
>>
>> Thank you for your fix.
>>
>> On 04/21/17 23:20, Benjamin Poirier wrote:
>> > Some statistics passed to ethtool are garbage because e1000e_get_stats64()
>> > doesn't write them, for example: tx_heartbeat_errors. This leaks kernel
>> > memory to userspace and confuses users.
>>
>> Could you please give specific examples to reproduce the issue? That way
>> your fix can also be tested.
>>
>
> Some fields in e1000_get_ethtool_stats()'s net_stats are not initialized
> by e1000e_get_stats64(). The structure is allocated on the stack,
> therefore, the value of those fields depends on previous stack content;
> that in turns depends on kernel version, compiler and previous execution
> path.
I agree.
I read over these code paths earlier today and came to the same exact
conclusion.
^ permalink raw reply
* [PATCH] net: hso: fix module unloading
From: Andreas Kemnade @ 2017-04-24 19:18 UTC (permalink / raw)
To: davem, joe, peter, linux-usb, netdev, linux-kernel,
Discussions about the Letux Kernel
Cc: Andreas Kemnade
keep tty driver until usb driver is unregistered
rmmod hso
produces traces like this without that:
[40261.645904] usb 2-2: new high-speed USB device number 2 using ehci-omap
[40261.854644] usb 2-2: New USB device found, idVendor=0af0, idProduct=8800
[40261.862609] usb 2-2: New USB device strings: Mfr=3, Product=2, SerialNumber=0
[40261.872772] usb 2-2: Product: Globetrotter HSUPA Modem
[40261.880279] usb 2-2: Manufacturer: Option N.V.
[40262.021270] hso 2-2:1.5: Not our interface
[40265.556945] hso: unloaded
[40265.559875] usbcore: deregistering interface driver hso
[40265.595947] Unable to handle kernel NULL pointer dereference at virtual address 00000033
[40265.604522] pgd = ecb14000
[40265.611877] [00000033] *pgd=00000000
[40265.617034] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[40265.622650] Modules linked in: hso(-) bnep bluetooth ipv6 arc4 twl4030_madc_hwmon wl18xx wlcore mac80211 cfg80211 snd_soc_simple_card snd_soc_simple_card_utils snd_soc_omap_twl4030 snd_soc_gtm601 generic_adc_battery extcon_gpio omap3_isp videobuf2_dma_contig videobuf2_memops wlcore_sdio videobuf2_v4l2 videobuf2_core ov9650 bmp280_i2c v4l2_common bmp280 bmg160_i2c bmg160_core at24 nvmem_core videodev bmc150_accel_i2c bmc150_magn_i2c media bmc150_accel_core tsc2007 bmc150_magn leds_tca6507 bno055 snd_soc_omap_mcbsp industrialio_triggered_buffer snd_soc_omap kfifo_buf snd_pcm_dmaengine gpio_twl4030 snd_soc_twl4030 twl4030_vibra twl4030_madc wwan_on_off ehci_omap pwm_bl pwm_omap_dmtimer panel_tpo_td028ttec1 encoder_opa362 connector_analog_tv omapdrm drm_kms_helper cfbfillrect syscopyarea cf
bimgblt sysfillrect
[40265.698211] sysimgblt fb_sys_fops cfbcopyarea drm omapdss usb_f_ecm g_ether usb_f_rndis u_ether libcomposite configfs omap2430 phy_twl4030_usb musb_hdrc twl4030_charger industrialio w2sg0004 twl4030_pwrbutton bq27xxx_battery w1_bq27000 omap_hdq [last unloaded: hso]
[40265.723175] CPU: 0 PID: 2701 Comm: rmmod Not tainted 4.11.0-rc6-letux+ #6
[40265.730346] Hardware name: Generic OMAP36xx (Flattened Device Tree)
[40265.736938] task: ecb81100 task.stack: ecb82000
[40265.741729] PC is at cdev_del+0xc/0x2c
[40265.745666] LR is at tty_unregister_device+0x40/0x50
[40265.750915] pc : [<c027472c>] lr : [<c04b3ecc>] psr: 600b0113
sp : ecb83ea8 ip : eca4f898 fp : 00000000
[40265.763000] r10: 00000000 r9 : 00000000 r8 : 00000001
[40265.768493] r7 : eca4f800 r6 : 00000003 r5 : 00000000 r4 : ffffffff
[40265.775360] r3 : c1458d54 r2 : 00000000 r1 : 00000004 r0 : ffffffff
[40265.782257] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
[40265.789764] Control: 10c5387d Table: acb14019 DAC: 00000051
[40265.795806] Process rmmod (pid: 2701, stack limit = 0xecb82218)
[40265.802062] Stack: (0xecb83ea8 to 0xecb84000)
[40265.806640] 3ea0: ec9e8100 c04b3ecc bf737378 ed5b7c00 00000003 bf7327ec
[40265.815277] 3ec0: eca4f800 00000000 ec9fd800 eca4f800 bf737070 bf7328bc eca4f820 c05a9a04
[40265.823883] 3ee0: eca4f820 00000000 00000001 eca4f820 ec9fd870 bf737070 eca4f854 ec9fd8a4
[40265.832519] 3f00: ecb82000 00000000 00000000 c04e6960 eca4f820 bf737070 bf737048 00000081
[40265.841125] 3f20: c01071e4 c04e6a60 ecb81100 bf737070 bf737070 c04e5d94 bf737020 c05a8f88
[40265.849731] 3f40: bf737100 00000800 7f5fa254 00000081 c01071e4 c01c4afc 00000000 006f7368
[40265.858367] 3f60: ecb815f4 00000000 c0cac9c4 c01071e4 ecb82000 00000000 00000000 c01512f4
[40265.866973] 3f80: ed5b3200 c01071e4 7f5fa220 7f5fa220 bea78ec9 0010711c 7f5fa220 7f5fa220
[40265.875579] 3fa0: bea78ec9 c0107040 7f5fa220 7f5fa220 7f5fa254 00000800 dd35b800 dd35b800
[40265.884216] 3fc0: 7f5fa220 7f5fa220 bea78ec9 00000081 bea78dcc 00000000 bea78bd8 00000000
[40265.892822] 3fe0: b6f70521 bea78b6c 7f5dd613 b6f70526 80070030 7f5fa254 ffffffff ffffffff
[40265.901458] [<c027472c>] (cdev_del) from [<c04b3ecc>] (tty_unregister_device+0x40/0x50)
[40265.909942] [<c04b3ecc>] (tty_unregister_device) from [<bf7327ec>] (hso_free_interface+0x80/0x144 [hso])
[40265.919982] [<bf7327ec>] (hso_free_interface [hso]) from [<bf7328bc>] (hso_disconnect+0xc/0x18 [hso])
[40265.929718] [<bf7328bc>] (hso_disconnect [hso]) from [<c05a9a04>] (usb_unbind_interface+0x84/0x200)
[40265.939239] [<c05a9a04>] (usb_unbind_interface) from [<c04e6960>] (device_release_driver_internal+0x138/0x1cc)
[40265.949798] [<c04e6960>] (device_release_driver_internal) from [<c04e6a60>] (driver_detach+0x60/0x6c)
[40265.959503] [<c04e6a60>] (driver_detach) from [<c04e5d94>] (bus_remove_driver+0x64/0x8c)
[40265.968017] [<c04e5d94>] (bus_remove_driver) from [<c05a8f88>] (usb_deregister+0x5c/0xb8)
[40265.976654] [<c05a8f88>] (usb_deregister) from [<c01c4afc>] (SyS_delete_module+0x160/0x1dc)
[40265.985443] [<c01c4afc>] (SyS_delete_module) from [<c0107040>] (ret_fast_syscall+0x0/0x1c)
[40265.994171] Code: c1458d54 e59f3020 e92d4010 e1a04000 (e5941034)
[40266.016693] ---[ end trace 9d5ac43c7e41075c ]---
Signed-off-by: Andreas Kemnade <andreas@kemnade.info>
---
drivers/net/usb/hso.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/usb/hso.c b/drivers/net/usb/hso.c
index 2e66340..b75e23f 100644
--- a/drivers/net/usb/hso.c
+++ b/drivers/net/usb/hso.c
@@ -3293,9 +3293,9 @@ static void __exit hso_exit(void)
pr_info("unloaded\n");
tty_unregister_driver(tty_drv);
- put_tty_driver(tty_drv);
/* deregister the usb driver */
usb_deregister(&hso_driver);
+ put_tty_driver(tty_drv);
}
/* Module definitions */
--
2.1.4
^ permalink raw reply related
* [PATCH] stmmac: Add support for SIMATIC IOT2000 platform
From: Jan Kiszka @ 2017-04-24 19:27 UTC (permalink / raw)
To: Giuseppe Cavallaro, Alexandre Torgue
Cc: netdev, Linux Kernel Mailing List, Sascha Weisenberger
The IOT2000 is industrial controller platform, derived from the Intel
Galileo Gen2 board. The variant IOT2020 comes with one LAN port, the
IOT2040 has two of them. They can be told apart based on the board asset
tag in the DMI table.
Based on patch by Sascha Weisenberger.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: Sascha Weisenberger <sascha.weisenberger@siemens.com>
---
drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
index 5c9e462276b9..de87c329fb5c 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
@@ -32,6 +32,7 @@
*/
struct stmmac_pci_dmi_data {
const char *name;
+ const char *asset_tag;
unsigned int func;
int phy_addr;
};
@@ -46,6 +47,7 @@ struct stmmac_pci_info {
static int stmmac_pci_find_phy_addr(struct stmmac_pci_info *info)
{
const char *name = dmi_get_system_info(DMI_BOARD_NAME);
+ const char *asset_tag = dmi_get_system_info(DMI_BOARD_ASSET_TAG);
unsigned int func = PCI_FUNC(info->pdev->devfn);
struct stmmac_pci_dmi_data *dmi;
@@ -57,7 +59,8 @@ static int stmmac_pci_find_phy_addr(struct stmmac_pci_info *info)
return 1;
for (dmi = info->dmi; dmi->name && *dmi->name; dmi++) {
- if (!strcmp(dmi->name, name) && dmi->func == func)
+ if (dmi->func == func && !strcmp(dmi->name, name) &&
+ (!dmi->asset_tag || !strcmp(dmi->asset_tag, asset_tag)))
return dmi->phy_addr;
}
@@ -142,6 +145,24 @@ static struct stmmac_pci_dmi_data quark_pci_dmi_data[] = {
.func = 6,
.phy_addr = 1,
},
+ {
+ .name = "SIMATIC IOT2000",
+ .asset_tag = "6ES7647-0AA00-0YA2",
+ .func = 6,
+ .phy_addr = 1,
+ },
+ {
+ .name = "SIMATIC IOT2000",
+ .asset_tag = "6ES7647-0AA00-1YA2",
+ .func = 6,
+ .phy_addr = 1,
+ },
+ {
+ .name = "SIMATIC IOT2000",
+ .asset_tag = "6ES7647-0AA00-1YA2",
+ .func = 7,
+ .phy_addr = 1,
+ },
{}
};
^ permalink raw reply related
* Re: [PATCH 2/2] ixgbe: add support for XDP_TX action
From: Jesper Dangaard Brouer @ 2017-04-24 19:29 UTC (permalink / raw)
To: John Fastabend; +Cc: brouer, jeffrey.t.kirsher, netdev
In-Reply-To: <20170424013135.8354.93583.stgit@john-Precision-Tower-5810>
On Sun, 23 Apr 2017 18:31:36 -0700
John Fastabend <john.fastabend@gmail.com> wrote:
> +static int ixgbe_xmit_xdp_ring(struct ixgbe_adapter *adapter,
> + struct xdp_buff *xdp)
> +{
> + struct ixgbe_ring *ring = adapter->xdp_ring[smp_processor_id()];
I was about to question whether is it always true that the array size
can match the number of CPUs in the system, but I can see later in
ixgbe_xdp_setup() that you reject XDP program if the system have more
CPUs that MAX_XDP_QUEUES.
> + struct ixgbe_tx_buffer *tx_buffer;
> + union ixgbe_adv_tx_desc *tx_desc;
> + u32 len, cmd_type;
> + dma_addr_t dma;
> + u16 i;
> +
> + len = xdp->data_end - xdp->data;
> +
> + if (unlikely(!ixgbe_desc_unused(ring)))
> + return IXGBE_XDP_CONSUMED;
> +
> + dma = dma_map_single(ring->dev, xdp->data, len, DMA_TO_DEVICE);
> + if (dma_mapping_error(ring->dev, dma))
> + return IXGBE_XDP_CONSUMED;
> +
> + /* record the location of the first descriptor for this packet */
> + tx_buffer = &ring->tx_buffer_info[ring->next_to_use];
> + tx_buffer->bytecount = len;
> + tx_buffer->gso_segs = 1;
> + tx_buffer->protocol = 0;
> +
> + i = ring->next_to_use;
> + tx_desc = IXGBE_TX_DESC(ring, i);
> +
> + dma_unmap_len_set(tx_buffer, len, len);
> + dma_unmap_addr_set(tx_buffer, dma, dma);
> + tx_buffer->data = xdp->data;
> + tx_desc->read.buffer_addr = cpu_to_le64(dma);
> +
> + /* put descriptor type bits */
> + cmd_type = IXGBE_ADVTXD_DTYP_DATA |
> + IXGBE_ADVTXD_DCMD_DEXT |
> + IXGBE_ADVTXD_DCMD_IFCS;
> + cmd_type |= len | IXGBE_TXD_CMD;
> + tx_desc->read.cmd_type_len = cpu_to_le32(cmd_type);
> + tx_desc->read.olinfo_status =
> + cpu_to_le32(len << IXGBE_ADVTXD_PAYLEN_SHIFT);
> +
> + /* Force memory writes to complete before letting h/w know there
> + * are new descriptors to fetch. (Only applicable for weak-ordered
> + * memory model archs, such as IA-64).
> + *
> + * We also need this memory barrier to make certain all of the
> + * status bits have been updated before next_to_watch is written.
> + */
> + wmb();
> +
> + /* set next_to_watch value indicating a packet is present */
> + i++;
> + if (i == ring->count)
> + i = 0;
> +
> + tx_buffer->next_to_watch = tx_desc;
> + ring->next_to_use = i;
> +
> + writel(i, ring->tail);
A tailptr write for every XDP_TX packet is not going be fast, but you
already mentioned that this is not optimal yet, so I guess you are aware.
> + return IXGBE_XDP_TX;
> +}
On Sun, 23 Apr 2017 18:31:36 -0700
John Fastabend <john.fastabend@gmail.com> wrote:
> @@ -9559,9 +9740,23 @@ static int ixgbe_xdp_setup(struct net_device *dev, struct bpf_prog *prog)
> return -EINVAL;
> }
>
> + if (nr_cpu_ids > MAX_XDP_QUEUES)
> + return -ENOMEM;
> +
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* Re: [PATCH 2/2] ixgbe: add support for XDP_TX action
From: John Fastabend @ 2017-04-24 19:35 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: jeffrey.t.kirsher, netdev
In-Reply-To: <20170424212921.36fdfff7@redhat.com>
On 17-04-24 12:29 PM, Jesper Dangaard Brouer wrote:
> On Sun, 23 Apr 2017 18:31:36 -0700
> John Fastabend <john.fastabend@gmail.com> wrote:
>
>> +static int ixgbe_xmit_xdp_ring(struct ixgbe_adapter *adapter,
>> + struct xdp_buff *xdp)
>> +{
>> + struct ixgbe_ring *ring = adapter->xdp_ring[smp_processor_id()];
>
> I was about to question whether is it always true that the array size
> can match the number of CPUs in the system, but I can see later in
> ixgbe_xdp_setup() that you reject XDP program if the system have more
> CPUs that MAX_XDP_QUEUES.
Yep.
[...]
>> +
>> + tx_buffer->next_to_watch = tx_desc;
>> + ring->next_to_use = i;
>> +
>> + writel(i, ring->tail);
>
> A tailptr write for every XDP_TX packet is not going be fast, but you
> already mentioned that this is not optimal yet, so I guess you are aware.
>
There is another patch on Jeff's tree to only kick the tail ptr once per
receive path invocation.
https://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/next-queue.git/commit/?h=dev-queue&id=24043a662d11e048de903e12bf86059844c207e2
That patch brings packet rates up to near line rate @ 64 bytes.
Thanks,
John
^ permalink raw reply
* Re: [PATCH v3 net] net: ipv6: regenerate host route if moved to gc list
From: David Ahern @ 2017-04-24 19:37 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev, dvyukov, andreyknvl, mmanning, kafai
In-Reply-To: <1493051946.6453.37.camel@edumazet-glaptop3.roam.corp.google.com>
On 4/24/17 10:39 AM, Eric Dumazet wrote:
>
> Very nice changelog !
Thanks. Given my aggressive brain cell recycling program, I needed to
write down the analysis.
>> diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
>> index 80ce478c4851..93f81d9cd85f 100644
>> --- a/net/ipv6/addrconf.c
>> +++ b/net/ipv6/addrconf.c
>> @@ -3271,14 +3271,25 @@ static void addrconf_gre_config(struct net_device *dev)
>> static int fixup_permanent_addr(struct inet6_dev *idev,
>> struct inet6_ifaddr *ifp)
>> {
>> - if (!ifp->rt) {
>> - struct rt6_info *rt;
>> + /* rt6i_ref == 0 means the host route was removed from the
>> + * FIB, for example, if 'lo' device is taken down. In that
>> + * case regenerate the host route.
>> + */
>> + if (!ifp->rt || !atomic_read(&ifp->rt->rt6i_ref)) {
>> + struct rt6_info *rt, *prev;
>>
>> rt = addrconf_dst_alloc(idev, &ifp->addr, false);
>> if (unlikely(IS_ERR(rt)))
>> return PTR_ERR(rt);
>>
>> + prev = ifp->rt;
>
> I would feel more comfortable if this was moved after the spin_lock() ?
That's what I had in v2; it reads better to me even if it is not
technically required (all changes to ifp->rt happen under rtnl).
Martin you agree?
I'll send a v3 tomorrow -- allow more time for other comments.
^ permalink raw reply
* Re: Get amount of fast retransmissions from TCP info
From: Neal Cardwell @ 2017-04-24 19:42 UTC (permalink / raw)
To: Lars Erik Storbukås; +Cc: LKML, Netdev
In-Reply-To: <CAF0XkCCFwxuyJ5bc4SUHjdDJb_E-CqYC-3k3nzi+0i3H9D2zPA@mail.gmail.com>
On Mon, Apr 24, 2017 at 3:11 PM, Lars Erik Storbukås
<storbukas.dev@gmail.com> wrote:
> I'm trying to get amount of congestion events in TCP caused by
> DUPACK's (fast retransmissions), and can't seem to find any variable
> in the TCP info struct which hold that value. There are three
> variables in the TCP info struct that seem to hold similar congestion
> values: __u8 tcpi_retransmits;__u32 tcpi_retrans; __u32
> tcpi_total_retrans;
>
> Does anyone have any pointers on how to find this value in the TCP code?
>
> Please CC me personally if answering this question. Any help is
> greatly appreciated.
[I'm cc-ing the netdev list.]
Do you need this per-socket? On a per-socket basis, I do not think
there are separate totals for fast retransmits and timeout
retransmits.
If a global number is good enough, then you can get that number from
the global network statistics. In "nstat" output they look like:
TcpExtTCPFastRetrans = packets sent in fast retransmit / fast recovery
TcpExtTCPSlowStartRetrans = packets sent in timeout recovery
It sounds like TcpExtTCPFastRetrans is what you are after.
Hope that helps,
neal
^ permalink raw reply
* Networking security fixes
From: Ben Hutchings @ 2017-04-24 19:47 UTC (permalink / raw)
To: David S. Miller; +Cc: stable, netdev
Please queue up these fixes for 4.10.y:
8605330aac5a tcp: fix SCM_TIMESTAMPING_OPT_STATS for normal skbs
4ef1b2869447 tcp: mark skbs with SCM_TIMESTAMPING_OPT_STATS
If I understand correctly, you are no longer sending Greg fixes for
4.4.y or older stable branches, so I'll send stable requests for those
older versions directly.
Please can you also send the pending fixes to Greg soon, as the
AF_PACKET issue is quite serious?
Ben.
--
Ben Hutchings
Software Developer, Codethink Ltd.
^ permalink raw reply
* Re: [PATCH 1/1] net: bcmgenet: fix incorrect return value checks
From: David Miller @ 2017-04-24 19:48 UTC (permalink / raw)
To: bianpan201602; +Cc: f.fainelli, netdev, linux-kernel, bianpan2016
In-Reply-To: <1492941665-17155-1-git-send-email-bianpan201602@163.com>
From: Pan Bian <bianpan201602@163.com>
Date: Sun, 23 Apr 2017 18:01:05 +0800
> From: Pan Bian <bianpan2016@163.com>
>
> Function platform_get_irq() will return a negative value on errors.
> However, in function bcmgenet_probe(), 0 is considered as a flag of
> error. This patch fixes the bug by checking whether the return value of
> platform_get_irq() is less than 0.
>
> Signed-off-by: Pan Bian <bianpan2016@163.com>
I'm definitely not confident enough to apply this.
On some platform zero IRQs are invalid.
There are also lots of pieces of code that check the return value ">
0" as success.
^ permalink raw reply
* Re: [PATCH 1/1] lwtunnel: check return value of nla_nest_start
From: David Miller @ 2017-04-24 19:51 UTC (permalink / raw)
To: bianpan2016
Cc: dsa, roopa, ast, david.lebrun, tom, rshearma, netdev,
linux-kernel
In-Reply-To: <1492928917-25628-1-git-send-email-bianpan2016@163.com>
From: Pan Bian <bianpan2016@163.com>
Date: Sun, 23 Apr 2017 14:28:37 +0800
> Function nla_nest_start() may return a NULL pointer on error. However,
> in function lwtunnel_fill_encap(), the return value of nla_nest_start()
> is not validated before it is used. This patch checks the return value
> of nla_nest_start() against NULL.
>
> Signed-off-by: Pan Bian <bianpan2016@163.com>
Applied.
^ permalink raw reply
* Re: [PATCH 1/1] tipc: check return value of nlmsg_new
From: David Miller @ 2017-04-24 19:51 UTC (permalink / raw)
To: bianpan2016; +Cc: jon.maloy, ying.xue, netdev, tipc-discussion, linux-kernel
In-Reply-To: <1492931359-25004-1-git-send-email-bianpan2016@163.com>
From: Pan Bian <bianpan2016@163.com>
Date: Sun, 23 Apr 2017 15:09:19 +0800
> Function nlmsg_new() will return a NULL pointer if there is no enough
> memory, and its return value should be checked before it is used.
> However, in function tipc_nl_node_get_monitor(), the validation of the
> return value of function nlmsg_new() is missed. This patch fixes the
> bug.
>
> Signed-off-by: Pan Bian <bianpan2016@163.com>
Applied.
^ permalink raw reply
* Re: [PATCH 1/1] wan: pc300too: abort path on failure
From: David Miller @ 2017-04-24 19:51 UTC (permalink / raw)
To: bianpan201602; +Cc: khc, netdev, linux-kernel, bianpan2016
In-Reply-To: <1492940315-1462-1-git-send-email-bianpan201602@163.com>
From: Pan Bian <bianpan201602@163.com>
Date: Sun, 23 Apr 2017 17:38:35 +0800
> From: Pan Bian <bianpan2016@163.com>
>
> In function pc300_pci_init_one(), on the ioremap error path, function
> pc300_pci_remove_one() is called to free the allocated memory. However,
> the path is not terminated, and the freed memory will be used later,
> resulting in use-after-free bugs. This path fixes the bug.
>
> Signed-off-by: Pan Bian <bianpan2016@163.com>
Applied.
^ permalink raw reply
* Re: [PATCH 1/1] net: bcmgenet: fix incorrect return value checks
From: Florian Fainelli @ 2017-04-24 19:52 UTC (permalink / raw)
To: David Miller, bianpan201602; +Cc: netdev, linux-kernel, bianpan2016
In-Reply-To: <20170424.154806.387657447648095027.davem@davemloft.net>
On 04/24/2017 12:48 PM, David Miller wrote:
> From: Pan Bian <bianpan201602@163.com>
> Date: Sun, 23 Apr 2017 18:01:05 +0800
>
>> From: Pan Bian <bianpan2016@163.com>
>>
>> Function platform_get_irq() will return a negative value on errors.
>> However, in function bcmgenet_probe(), 0 is considered as a flag of
>> error. This patch fixes the bug by checking whether the return value of
>> platform_get_irq() is less than 0.
>>
>> Signed-off-by: Pan Bian <bianpan2016@163.com>
>
> I'm definitely not confident enough to apply this.
>
> On some platform zero IRQs are invalid.
>
> There are also lots of pieces of code that check the return value ">
> 0" as success.
>
I don't think we are fixing any real issue by applying this patch, but I
will do a check on ARM, ARM64 and MIPS where this driver is used to see
if it is even remotely possible to have a 0 IRQ.
--
Florian
^ permalink raw reply
* Re: [PATCH] net: bridge: suppress broadcast when multicast flood is disabled
From: Nikolay Aleksandrov @ 2017-04-24 19:52 UTC (permalink / raw)
To: Mike Manning, netdev; +Cc: David S. Miller, roopa
In-Reply-To: <1493042944-4005-1-git-send-email-mmanning@brocade.com>
On 24/04/17 17:09, Mike Manning wrote:
> Flood suppression for packets that are not unicast needs to be handled
> consistently by also not flooding broadcast packets. As broadcast is a
> special case of multicast, the same kernel parameter should be used to
> suppress flooding for both of these packet types.
>
> Fixes: b6cb5ac8331b ("net: bridge: add per-port multicast flood flag")
> Cc: Nikolay Aleksandrov <nikolay@cumulusnetworks.com>
> Signed-off-by: Mike Manning <mmanning@brocade.com>
> ---
> net/bridge/br_forward.c | 17 ++++++++++-------
> 1 file changed, 10 insertions(+), 7 deletions(-)
>
I do not agree that this is a bug fix, the behaviour was intentional and is close to how HW
handles this flag. It has been like that for a few releases and changing it may impact setups
that use the flag since up until now they've seen the broadcast but not multicast packets and
suddenly their broadcast will stop.
I think it would be better to introduce a third flag for bcast in net-next and use that to
filter it since that would give us the ability to program HW that can distinguish these
and have both options available, moreover it will not break any user setups relying on
the current flag behaviour and we have such setups.
Thanks,
Nik
^ permalink raw reply
* RE:PATCH drivers:net:cris/eth_v10: alternate string char arrary
From: Karim Eshapa @ 2017-04-24 19:54 UTC (permalink / raw)
To: davem
Cc: felipe.balbi, paul.gortmaker, mugunthanvnm, jarod, fw, netdev,
linux-kernel, Karim Eshapa
In-Reply-To: <1493056179-6460-1-git-send-email-karim.eshapa@gmail.com>
On Mon, 24 Apr 2017 14:18:58 -0400 (EDT), David Miller wrote:
> Mon, 24 Apr 2017 19:49:39 +0200, Karim Eshapa wrote:
>>
>> static char pointer creates two variables in final assembly.
>> static string and pointer to it according to
>> Jeff Garzik janitors TODO.
>
> Instead of trusting some document written more than 10 years ago on
> the internet, why don't you build the source file in question and take
> a look at what actually happens?
>
I've just dumped the assembly and symbols. I didn't find
extraordinary difference between the two strings.
sorry for that, but why is that still there in
TODO kerneljanitors at kernelnewbies.
Thanks,
Karim
^ permalink raw reply
* Re: [PATCH 5/7] IB/hfi1: use pcie_flr instead of duplicating it
From: Dennis Dalessandro @ 2017-04-24 20:00 UTC (permalink / raw)
To: Christoph Hellwig, Byczkowski, Jakub
Cc: Bjorn Helgaas, Cabiddu, Giovanni, Benedetto, Salvatore,
Marciniszyn, Mike, Derek Chickles, Satanand Burla, Felix Manlunas,
Raghu Vatsavayi, Kirsher, Jeffrey T,
linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, qat-linux,
linux-crypto-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20170424143507.GA28812-jcswGhMUV9g@public.gmane.org>
On 04/24/2017 10:35 AM, Christoph Hellwig wrote:
> On Mon, Apr 24, 2017 at 02:16:31PM +0000, Byczkowski, Jakub wrote:
>> Tested-by: Jakub Byczkowski <jakub.byczkowski-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> Are you (and Doug) ok with queueing this up in the PCI tree?
We are fine however Doug wants to handle it.
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [pull request][net 0/7] Mellanox, mlx5 fixes 2017-04-22
From: David Miller @ 2017-04-24 20:01 UTC (permalink / raw)
To: saeedm; +Cc: netdev
In-Reply-To: <20170423100802.27630-1-saeedm@mellanox.com>
From: Saeed Mahameed <saeedm@mellanox.com>
Date: Sun, 23 Apr 2017 13:07:55 +0300
> This series contains some mlx5 fixes for net.
>
> For your convenience, the series doesn't introduce any conflict with
> the ongoing net-next pull request.
>
> Please pull and let me know if there's any problem.
Pulled.
> For -stable:
> ("net/mlx5: E-Switch, Correctly deal with inline mode on ConnectX-5") kernels >= 4.10
> ("net/mlx5e: Fix ETHTOOL_GRXCLSRLALL handling") kernels >= 4.8
> ("net/mlx5e: Fix small packet threshold") kernels >= 4.7
> ("net/mlx5: Fix driver load bad flow when having fw initializing timeout") kernels >= 4.4
Queued up, thanks.
^ permalink raw reply
* Re: PATCH drivers:net:cris/eth_v10: alternate string char arrary
From: David Miller @ 2017-04-24 20:04 UTC (permalink / raw)
To: karim.eshapa
Cc: felipe.balbi, paul.gortmaker, mugunthanvnm, jarod, fw, netdev,
linux-kernel
In-Reply-To: <1493063682-8560-1-git-send-email-karim.eshapa@gmail.com>
From: Karim Eshapa <karim.eshapa@gmail.com>
Date: Mon, 24 Apr 2017 21:54:42 +0200
> On Mon, 24 Apr 2017 14:18:58 -0400 (EDT), David Miller wrote:
>> Mon, 24 Apr 2017 19:49:39 +0200, Karim Eshapa wrote:
>>>
>>> static char pointer creates two variables in final assembly.
>>> static string and pointer to it according to
>>> Jeff Garzik janitors TODO.
>>
>> Instead of trusting some document written more than 10 years ago on
>> the internet, why don't you build the source file in question and take
>> a look at what actually happens?
>>
>
> I've just dumped the assembly and symbols. I didn't find
> extraordinary difference between the two strings.
> sorry for that, but why is that still there in
> TODO kerneljanitors at kernelnewbies.
Because nobody is updating the documentation.
^ permalink raw reply
* Re: Networking security fixes
From: David Miller @ 2017-04-24 20:07 UTC (permalink / raw)
To: ben.hutchings; +Cc: stable, netdev
In-Reply-To: <1493063249.10415.78.camel@codethink.co.uk>
From: Ben Hutchings <ben.hutchings@codethink.co.uk>
Date: Mon, 24 Apr 2017 20:47:29 +0100
> Please queue up these fixes for 4.10.y:
>
> 8605330aac5a tcp: fix SCM_TIMESTAMPING_OPT_STATS for normal skbs
> 4ef1b2869447 tcp: mark skbs with SCM_TIMESTAMPING_OPT_STATS
Ok, queued up, thanks.
> Please can you also send the pending fixes to Greg soon, as the
> AF_PACKET issue is quite serious?
I will get to it as soon as I can.
^ permalink raw reply
* Re: [PATCH 1/1] qlcnic: fix unchecked return value
From: David Miller @ 2017-04-24 20:11 UTC (permalink / raw)
To: bianpan201602
Cc: harish.patil, manish.chopra, Dept-GELinuxNICDev, netdev,
linux-kernel, bianpan2016
In-Reply-To: <1492949044-23580-1-git-send-email-bianpan201602@163.com>
aFrom: Pan Bian <bianpan201602@163.com>
Date: Sun, 23 Apr 2017 20:04:04 +0800
> From: Pan Bian <bianpan2016@163.com>
>
> Function pci_find_ext_capability() may return 0, which is an invalid
> address. In function qlcnic_sriov_virtid_fn(), its return value is used
> without validation. This may result in invalid memory access bugs. This
> patch fixes the bug.
>
> Signed-off-by: Pan Bian <bianpan2016@163.com>
Applied.
^ permalink raw reply
* [PATCH net-next] bpf: make bpf_xdp_adjust_head support mandatory
From: Daniel Borkmann @ 2017-04-24 20:14 UTC (permalink / raw)
To: davem; +Cc: ast, netdev, Daniel Borkmann
Now that also the last in-tree user of the xdp_adjust_head bit has
been removed, we can remove the flag from struct bpf_prog altogether.
This, at the same time, also makes sure that any future driver for
XDP comes with bpf_xdp_adjust_head() support right away.
A rejection based on this flag would also mean that tail calls
couldn't be used with such driver as per c2002f983767 ("bpf: fix
checking xdp_adjust_head on tail calls") fix, thus lets not allow
for it in the first place.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/filter.h | 3 +--
kernel/bpf/verifier.c | 3 ---
2 files changed, 1 insertion(+), 5 deletions(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 511fe91..9a7786d 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -413,8 +413,7 @@ struct bpf_prog {
locked:1, /* Program image locked? */
gpl_compatible:1, /* Is filter GPL compatible? */
cb_access:1, /* Is control block accessed? */
- dst_needed:1, /* Do we need dst entry? */
- xdp_adjust_head:1; /* Adjusting pkt head? */
+ dst_needed:1; /* Do we need dst entry? */
kmemcheck_bitfield_end(meta);
enum bpf_prog_type type; /* Type of BPF program */
u32 len; /* Number of filter blocks */
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index ca15cf2..6f8b6ed 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3346,8 +3346,6 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env)
prog->dst_needed = 1;
if (insn->imm == BPF_FUNC_get_prandom_u32)
bpf_user_rnd_init_once();
- if (insn->imm == BPF_FUNC_xdp_adjust_head)
- prog->xdp_adjust_head = 1;
if (insn->imm == BPF_FUNC_tail_call) {
/* If we tail call into other programs, we
* cannot make any assumptions since they can
@@ -3355,7 +3353,6 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env)
* the program array.
*/
prog->cb_access = 1;
- prog->xdp_adjust_head = 1;
/* mark bpf_tail_call as different opcode to avoid
* conditional branch in the interpeter for every normal
--
1.9.3
^ permalink raw reply related
* EXTREMELY IMPORTANT
From: Ms. Katherine @ 2017-04-24 18:59 UTC (permalink / raw)
To: Recipients
Dear Beloved, Sorry for the inconvenience, I am getting in touch with you regarding an extremely important and urgent matter, Please I need your urgent assistance and idea to finish up a project (Orphanage Home) Due to my health condition, Everything is available to finish up the project, All I need is your idea and trust.
Sehr geehrte Geliebte, traurig für die Unannehmlichkeiten, ich bin in Kontakt mit Ihnen über eine äußerst wichtige und dringende Angelegenheit, Bitte brauche ich Ihre dringende Hilfe und Idee, um ein Projekt zu beenden (Orphanage Home) Wegen meines Gesundheitszustandes ist alles verfügbar Beenden Sie das Projekt, Alles was ich brauche ist Ihre Idee und Vertrauen.
Please contact me for more details.
Contact Email: kathrynnmison@gmail.com
Thanks
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox