* [pci PATCH v8 3/4] nvme: Migrate over to unmanaged SR-IOV support
From: Alexander Duyck @ 2018-04-20 16:31 UTC (permalink / raw)
To: bhelgaas, alexander.h.duyck, linux-pci
Cc: virtio-dev, kvm, netdev, dan.daly, linux-kernel, linux-nvme,
keith.busch, netanel, ddutile, mheyne, liang-min.wang,
mark.d.rustad, dwmw2, hch, dwmw
In-Reply-To: <20180420162633.46077.49012.stgit@ahduyck-green-test.jf.intel.com>
Instead of implementing our own version of a SR-IOV configuration stub in
the nvme driver we can just reuse the existing
pci_sriov_configure_simple function.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
---
v5: Replaced call to pci_sriov_configure_unmanaged with
pci_sriov_configure_simple
v6: Dropped "#ifdef" checks for IOV wrapping sriov_configure definition
v7: No code change, added Reviewed-by
drivers/nvme/host/pci.c | 20 +-------------------
1 file changed, 1 insertion(+), 19 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index b6f43b7..ad85cf35 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2581,24 +2581,6 @@ static void nvme_remove(struct pci_dev *pdev)
nvme_put_ctrl(&dev->ctrl);
}
-static int nvme_pci_sriov_configure(struct pci_dev *pdev, int numvfs)
-{
- int ret = 0;
-
- if (numvfs == 0) {
- if (pci_vfs_assigned(pdev)) {
- dev_warn(&pdev->dev,
- "Cannot disable SR-IOV VFs while assigned\n");
- return -EPERM;
- }
- pci_disable_sriov(pdev);
- return 0;
- }
-
- ret = pci_enable_sriov(pdev, numvfs);
- return ret ? ret : numvfs;
-}
-
#ifdef CONFIG_PM_SLEEP
static int nvme_suspend(struct device *dev)
{
@@ -2717,7 +2699,7 @@ static void nvme_error_resume(struct pci_dev *pdev)
.driver = {
.pm = &nvme_dev_pm_ops,
},
- .sriov_configure = nvme_pci_sriov_configure,
+ .sriov_configure = pci_sriov_configure_simple,
.err_handler = &nvme_err_handler,
};
^ permalink raw reply related
* [pci PATCH v8 4/4] pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
From: Alexander Duyck @ 2018-04-20 16:31 UTC (permalink / raw)
To: bhelgaas, alexander.h.duyck, linux-pci
Cc: virtio-dev, kvm, netdev, dan.daly, linux-kernel, linux-nvme,
keith.busch, netanel, ddutile, mheyne, liang-min.wang,
mark.d.rustad, dwmw2, hch, dwmw
In-Reply-To: <20180420162633.46077.49012.stgit@ahduyck-green-test.jf.intel.com>
Add a new driver called "pci-pf-stub" to act as a "white-list" for PF
devices that provide no other functionality other then acting as a means of
allocating a set of VFs. For now I only have one example ID provided by
Amazon in terms of devices that require this functionality. The general
idea is that in the future we will see other devices added as vendors come
up with devices where the PF is more or less just a lightweight shim used
to allocate VFs.
Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
---
v6: New driver to address concerns about Amazon devices left unsupported
v7: Dropped pci_id table explanation from pci-pf-stub driver
drivers/pci/Kconfig | 12 ++++++++++
drivers/pci/Makefile | 2 ++
drivers/pci/pci-pf-stub.c | 54 +++++++++++++++++++++++++++++++++++++++++++++
include/linux/pci_ids.h | 2 ++
4 files changed, 70 insertions(+)
create mode 100644 drivers/pci/pci-pf-stub.c
diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index 34b56a8..cdef2a2 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -71,6 +71,18 @@ config PCI_STUB
When in doubt, say N.
+config PCI_PF_STUB
+ tristate "PCI PF Stub driver"
+ depends on PCI
+ depends on PCI_IOV
+ help
+ Say Y or M here if you want to enable support for devices that
+ require SR-IOV support, while at the same time the PF itself is
+ not providing any actual services on the host itself such as
+ storage or networking.
+
+ When in doubt, say N.
+
config XEN_PCIDEV_FRONTEND
tristate "Xen PCI Frontend"
depends on PCI && X86 && XEN
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index 9419709..4e133d3 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -43,6 +43,8 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o
obj-$(CONFIG_PCI_STUB) += pci-stub.o
+obj-$(CONFIG_PCI_PF_STUB) += pci-pf-stub.o
+
obj-$(CONFIG_PCI_ECAM) += ecam.o
obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
diff --git a/drivers/pci/pci-pf-stub.c b/drivers/pci/pci-pf-stub.c
new file mode 100644
index 0000000..9d5fdf2
--- /dev/null
+++ b/drivers/pci/pci-pf-stub.c
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: GPL-2.0
+/* pci-pf-stub - simple stub driver for PCI SR-IOV PF device
+ *
+ * This driver is meant to act as a "white-list" for devices that provde
+ * SR-IOV functionality while at the same time not actually needing a
+ * driver of their own.
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+
+/**
+ * pci_pf_stub_white_list - White list of devices to bind pci-pf-stub onto
+ *
+ * This table provides the list of IDs this driver is supposed to bind
+ * onto. You could think of this as a list of "quirked" devices where we
+ * are adding support for SR-IOV here since there are no other drivers
+ * that they would be running under.
+ */
+static const struct pci_device_id pci_pf_stub_white_list[] = {
+ { PCI_VDEVICE(AMAZON, 0x0053) },
+ /* required last entry */
+ { 0 }
+};
+MODULE_DEVICE_TABLE(pci, pci_pf_stub_white_list);
+
+static int pci_pf_stub_probe(struct pci_dev *dev,
+ const struct pci_device_id *id)
+{
+ pci_info(dev, "claimed by pci-pf-stub\n");
+ return 0;
+}
+
+static struct pci_driver pf_stub_driver = {
+ .name = "pci-pf-stub",
+ .id_table = pci_pf_stub_white_list,
+ .probe = pci_pf_stub_probe,
+ .sriov_configure = pci_sriov_configure_simple,
+};
+
+static int __init pci_pf_stub_init(void)
+{
+ return pci_register_driver(&pf_stub_driver);
+}
+
+static void __exit pci_pf_stub_exit(void)
+{
+ pci_unregister_driver(&pf_stub_driver);
+}
+
+module_init(pci_pf_stub_init);
+module_exit(pci_pf_stub_exit);
+
+MODULE_LICENSE("GPL");
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index a637a7d..62dab14 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2549,6 +2549,8 @@
#define PCI_VENDOR_ID_CIRCUITCO 0x1cc8
#define PCI_SUBSYSTEM_ID_CIRCUITCO_MINNOWBOARD 0x0001
+#define PCI_VENDOR_ID_AMAZON 0x1d0f
+
#define PCI_VENDOR_ID_TEKRAM 0x1de1
#define PCI_DEVICE_ID_TEKRAM_DC290 0xdc29
^ permalink raw reply related
* Re: [PATCH 0/8] arm: renesas: Change platform dependency to ARCH_RENESAS
From: Mark Brown @ 2018-04-20 16:48 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: alsa-devel, Kuninori Morimoto, Catalin Marinas, Will Deacon,
Liam Girdwood, Laurent Pinchart, devel, Mauro Carvalho Chehab,
Vinod Koul, Magnus Damm, Russell King, linux-media, Arnd Bergmann,
Simon Horman, Dan Williams, Jaroslav Kysela, linux-arm-kernel,
Sergei Shtylyov, Greg Kroah-Hartman, Takashi Iwai, linux-kernel,
linux-renesas-soc, netdev, dmaengine
In-Reply-To: <1524230914-10175-1-git-send-email-geert+renesas@glider.be>
[-- Attachment #1.1: Type: text/plain, Size: 460 bytes --]
On Fri, Apr 20, 2018 at 03:28:26PM +0200, Geert Uytterhoeven wrote:
> The first 6 patches can be applied independently by subsystem
> maintainers.
> The last two patches depend on the first 6 patches, and are thus marked
> RFC.
Would it not make sense to try to apply everything en masse rather than
delaying? I'm happy to apply the subsystem stuff but if it gets things
done quicker or more efficiently I'm also happy to have the lot merged
as one series.
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
[-- Attachment #2: Type: text/plain, Size: 169 bytes --]
_______________________________________________
devel mailing list
devel@linuxdriverproject.org
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
^ permalink raw reply
* [RFC iproute 0/5] print_uint issues
From: Stephen Hemminger @ 2018-04-20 17:15 UTC (permalink / raw)
To: alin.nastac; +Cc: netdev, Stephen Hemminger
There are several issues with implicit conversions in json
handling print_uint function which were first seen and patches
proposed by Alin Nastac. This is my swipe at them.
Final version will be some combination of his patches and
some of this.
Spotted some more changing print_uint into a macro,
but that got ugly.
Stephen Hemminger (5):
ipneigh: fix missing format specifier
json: make json print_uint take unsigned int
flower: use 16 bit format where possible
tcp_metrics: use print_luint
mroute: use print_uint64
include/json_print.h | 16 +++++++++++++---
ip/ipmroute.c | 10 +++++-----
ip/ipneigh.c | 2 +-
ip/tcp_metrics.c | 2 +-
lib/json_print.c | 33 ++++++++++++++++++++++++++++++---
tc/f_flower.c | 2 +-
6 files changed, 51 insertions(+), 14 deletions(-)
--
2.17.0
^ permalink raw reply
* [RFC iproute 1/5] ipneigh: fix missing format specifier
From: Stephen Hemminger @ 2018-04-20 17:15 UTC (permalink / raw)
To: alin.nastac; +Cc: netdev, Stephen Hemminger
In-Reply-To: <20180420171519.8028-1-stephen@networkplumber.org>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
ip/ipneigh.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ip/ipneigh.c b/ip/ipneigh.c
index 4748381701e4..bd6e5c5e18ca 100644
--- a/ip/ipneigh.c
+++ b/ip/ipneigh.c
@@ -204,7 +204,7 @@ static void print_cacheinfo(const struct nda_cacheinfo *ci)
print_uint(PRINT_ANY, "used", " used %u", ci->ndm_used / hz);
print_uint(PRINT_ANY, "confirmed", "/%u", ci->ndm_confirmed / hz);
- print_uint(PRINT_ANY, "updated", "/u", ci->ndm_updated / hz);
+ print_uint(PRINT_ANY, "updated", "/%u", ci->ndm_updated / hz);
}
static void print_neigh_state(unsigned int nud)
--
2.17.0
^ permalink raw reply related
* [RFC iproute 2/5] json: make json print_uint take unsigned int
From: Stephen Hemminger @ 2018-04-20 17:15 UTC (permalink / raw)
To: alin.nastac; +Cc: netdev, Stephen Hemminger
In-Reply-To: <20180420171519.8028-1-stephen@networkplumber.org>
It is a source of possible bugs that json print handler print_uint
was doing implict conversion to 64 bit than printing with the
format specifier which often had only a unsigned format value.
Instead introduce wider range of print stubs for unsigned
integer types. No color versions of these necessary.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
include/json_print.h | 16 +++++++++++++---
lib/json_print.c | 33 ++++++++++++++++++++++++++++++---
2 files changed, 43 insertions(+), 6 deletions(-)
diff --git a/include/json_print.h b/include/json_print.h
index 2ca7830adbd6..3d400ecf8f50 100644
--- a/include/json_print.h
+++ b/include/json_print.h
@@ -59,12 +59,22 @@ _PRINT_FUNC(int, int);
_PRINT_FUNC(bool, bool);
_PRINT_FUNC(null, const char*);
_PRINT_FUNC(string, const char*);
-_PRINT_FUNC(uint, uint64_t);
-_PRINT_FUNC(hu, unsigned short);
_PRINT_FUNC(hex, unsigned int);
_PRINT_FUNC(0xhex, unsigned int);
-_PRINT_FUNC(lluint, unsigned long long int);
_PRINT_FUNC(float, double);
#undef _PRINT_FUNC
+#define _PRINT_FUNC(type_name, type) \
+ void print_##type_name(enum output_type t, \
+ const char *key, \
+ const char *fmt, \
+ type value);
+
+_PRINT_FUNC(hu, unsigned short);
+_PRINT_FUNC(uint, unsigned int);
+_PRINT_FUNC(luint, unsigned long int);
+_PRINT_FUNC(uint64, uint64_t);
+_PRINT_FUNC(lluint, unsigned long long int);
+#undef _PRINT_FUNC
+
#endif /* _JSON_PRINT_H_ */
diff --git a/lib/json_print.c b/lib/json_print.c
index bda7293350c3..696d8c01d3e6 100644
--- a/lib/json_print.c
+++ b/lib/json_print.c
@@ -116,12 +116,39 @@ void close_json_array(enum output_type type, const char *str)
} \
}
_PRINT_FUNC(int, int);
-_PRINT_FUNC(hu, unsigned short);
-_PRINT_FUNC(uint, uint64_t);
-_PRINT_FUNC(lluint, unsigned long long int);
_PRINT_FUNC(float, double);
#undef _PRINT_FUNC
+static void json_print_uint64(const char *key, uint64_t val)
+{
+ if (key)
+ jsonw_uint_field(_jw, key, val);
+ else
+ jsonw_uint(_jw, val);
+}
+
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wformat-nonliteral"
+#define PRINT_FUNC(type_name, type) \
+ void print_##type_name(enum output_type t, \
+ const char *key, \
+ const char *fmt, \
+ type value) \
+ { \
+ if (_IS_JSON_CONTEXT(t)) \
+ json_print_uint64(key, value); \
+ else if (_IS_FP_CONTEXT(t)) \
+ printf(fmt, value); \
+ }
+
+PRINT_FUNC(hu, unsigned short);
+PRINT_FUNC(uint, unsigned int);
+PRINT_FUNC(lluint, unsigned long long int);
+PRINT_FUNC(luint, unsigned long int);
+PRINT_FUNC(uint64, uint64_t);
+#undef PRINT_FUNC
+#pragma GCC diagnostic pop
+
void print_color_string(enum output_type type,
enum color_attr color,
const char *key,
--
2.17.0
^ permalink raw reply related
* [RFC iproute 3/5] flower: use 16 bit format where possible
From: Stephen Hemminger @ 2018-04-20 17:15 UTC (permalink / raw)
To: alin.nastac; +Cc: netdev, Stephen Hemminger
In-Reply-To: <20180420171519.8028-1-stephen@networkplumber.org>
Should use print_hu not print_uint for 16 bit value.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
tc/f_flower.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tc/f_flower.c b/tc/f_flower.c
index 9d4bfd2f808b..ba8eb66cdd11 100644
--- a/tc/f_flower.c
+++ b/tc/f_flower.c
@@ -1234,7 +1234,7 @@ static void flower_print_port(char *name, struct rtattr *attr)
return;
sprintf(namefrm,"\n %s %%u", name);
- print_uint(PRINT_ANY, name, namefrm, rta_getattr_be16(attr));
+ print_hu(PRINT_ANY, name, namefrm, rta_getattr_be16(attr));
}
static void flower_print_tcp_flags(char *name, struct rtattr *flags_attr,
--
2.17.0
^ permalink raw reply related
* [RFC iproute 4/5] tcp_metrics: use print_luint
From: Stephen Hemminger @ 2018-04-20 17:15 UTC (permalink / raw)
To: alin.nastac; +Cc: netdev, Stephen Hemminger
In-Reply-To: <20180420171519.8028-1-stephen@networkplumber.org>
Metrics are long unsigned, so use correct print function.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
ip/tcp_metrics.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ip/tcp_metrics.c b/ip/tcp_metrics.c
index 72dc980c92a6..3b82a4e1d2f2 100644
--- a/ip/tcp_metrics.c
+++ b/ip/tcp_metrics.c
@@ -139,7 +139,7 @@ static void print_tcp_metrics(struct rtattr *a)
print_uint(PRINT_JSON, name, NULL, val);
print_string(PRINT_FP, NULL, " %s ", name);
- print_uint(PRINT_FP, NULL, "%lu", val);
+ print_luint(PRINT_FP, NULL, "%lu", val);
}
if (rtt) {
--
2.17.0
^ permalink raw reply related
* [RFC iproute 5/5] mroute: use print_uint64
From: Stephen Hemminger @ 2018-04-20 17:15 UTC (permalink / raw)
To: alin.nastac; +Cc: netdev, Stephen Hemminger
In-Reply-To: <20180420171519.8028-1-stephen@networkplumber.org>
The values from multicast stats are 64 bit always.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
ip/ipmroute.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/ip/ipmroute.c b/ip/ipmroute.c
index 59c5b7718e18..eb6b2816324f 100644
--- a/ip/ipmroute.c
+++ b/ip/ipmroute.c
@@ -182,14 +182,14 @@ int print_mroute(const struct sockaddr_nl *who, struct nlmsghdr *n, void *arg)
struct rta_mfc_stats *mfcs = RTA_DATA(tb[RTA_MFC_STATS]);
print_string(PRINT_FP, NULL, "%s", _SL_);
- print_uint(PRINT_ANY, "packets", " %"PRIu64" packets,",
+ print_uint64(PRINT_ANY, "packets", " %"PRIu64" packets,",
mfcs->mfcs_packets);
- print_uint(PRINT_ANY, "bytes", " %"PRIu64" bytes", mfcs->mfcs_bytes);
+ print_uint64(PRINT_ANY, "bytes", " %"PRIu64" bytes", mfcs->mfcs_bytes);
if (mfcs->mfcs_wrong_if)
- print_uint(PRINT_ANY, "wrong_if",
- ", %"PRIu64" arrived on wrong iif.",
- mfcs->mfcs_wrong_if);
+ print_uint64(PRINT_ANY, "wrong_if",
+ ", %"PRIu64" arrived on wrong iif.",
+ mfcs->mfcs_wrong_if);
}
if (show_stats && tb[RTA_EXPIRES]) {
--
2.17.0
^ permalink raw reply related
* Re: [PATCH net-next 0/5] virtio-net: Add SCTP checksum offload support
From: Marcelo Ricardo Leitner @ 2018-04-20 17:22 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Vlad Yasevich, Vladislav Yasevich, netdev, linux-sctp,
virtualization, jasowang, nhorman
In-Reply-To: <20180418162633-mutt-send-email-mst@kernel.org>
On Wed, Apr 18, 2018 at 05:06:46PM +0300, Michael S. Tsirkin wrote:
> On Tue, Apr 17, 2018 at 04:35:18PM -0400, Vlad Yasevich wrote:
> > On 04/02/2018 10:47 AM, Marcelo Ricardo Leitner wrote:
> > > On Mon, Apr 02, 2018 at 09:40:01AM -0400, Vladislav Yasevich wrote:
> > >> Now that we have SCTP offload capabilities in the kernel, we can add
> > >> them to virtio as well. First step is SCTP checksum.
> > >
> > > Thanks.
> > >
> > >> As for GSO, the way sctp GSO is currently implemented buys us nothing
> > >> in added support to virtio. To add true GSO, would require a lot of
> > >> re-work inside of SCTP and would require extensions to the virtio
> > >> net header to carry extra sctp data.
> > >
> > > Can you please elaborate more on this? Is this because SCTP GSO relies
> > > on the gso skb format for knowing how to segment it instead of having
> > > a list of sizes?
> > >
> >
> > it's mainly because all the true segmentation, placing data into chunks,
> > has already happened. All that GSO does is allow for higher bundling
> > rate between VMs. If that is all SCTP GSO ever going to do, that fine,
> > but the goal is to do real GSO eventually and potentially reduce the
> > amount of memory copying we are doing.
> > If we do that, any current attempt at GSO in virtio would have to be
> > depricated and we'd need GSO2 or something like that.
>
> Batching helps virtualization *a lot* though.
Yep. The results posted by Xin in the other email give good insights
on it.
> Are there actual plans for GSO2? Is it just for SCTP?
No plans. In this context, at least, yes, just for SCTP.
It was a supposition in case we start doing a different GSO for SCTP,
one more like what we have for TCP.
Currently, as the SCTP GSO code doesn't leave the system, we can
update it if we want. But by the moment we add support for it in
virtio, we will have to be backwards compatible if we end up doing
SCTP GSO differently.
But again, I don't think such approach for SCTP GSO would be neither
feasible or worth. The complexity for it, to work across stream
schedules and late TSN allocation, would do more harm then good IMO.
>
> >
> > This is why, after doing the GSO support, I decided not to include it.
> >
> > -vlad
> > > Marcelo
> > >
^ permalink raw reply
* Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
From: Randy Dunlap @ 2018-04-20 17:23 UTC (permalink / raw)
To: Alexander Duyck, bhelgaas, linux-pci
Cc: virtio-dev, kvm, netdev, dan.daly, linux-kernel, linux-nvme,
keith.busch, netanel, ddutile, mheyne, liang-min.wang,
mark.d.rustad, dwmw2, hch, dwmw
In-Reply-To: <20180420162633.46077.49012.stgit@ahduyck-green-test.jf.intel.com>
On 04/20/18 09:28, Alexander Duyck wrote:
> This series is meant to add support for SR-IOV on devices when the VFs are
> not managed by the kernel. Examples of recent patches attempting to do this
> include:
> virto - https://patchwork.kernel.org/patch/10241225/
> pci-stub - https://patchwork.kernel.org/patch/10109935/
> vfio - https://patchwork.kernel.org/patch/10103353/
> uio - https://patchwork.kernel.org/patch/9974031/
Hi,
Somewhere in this patch series it would be nice to tell us what the heck
a "PF" is. :)
Thanks.
> Since this is quickly blowing up into a multi-driver problem it is probably
> best to implement this solution as generically as possible.
>
> This series is an attempt to do that. What we do with this patch set is
> provide a generic framework to enable SR-IOV in the case that the PF driver
> doesn't support managing the VFs itself.
>
> I based my patch set originally on the patch by Mark Rustad but there isn't
> much left after going through and cleaning out the bits that were no longer
> needed, and after incorporating the feedback from David Miller. At this point
> the only items to be fully reused was his patch description which is now
> present in patch 3 of the set.
>
> This solution is limited in scope to just adding support for devices that
> provide no functionality for SR-IOV other than allocating the VFs by
> calling pci_enable_sriov. Previous sets had included patches for VFIO, but
> for now I am dropping that as the scope of that work is larger then I
> think I can take on at this time.
>
> v2: Reduced scope back to just virtio_pci and vfio-pci
> Broke into 3 patch set from single patch
> Changed autoprobe behavior to always set when num_vfs is set non-zero
> v3: Updated Documentation to clarify when sriov_unmanaged_autoprobe is used
> Wrapped vfio_pci_sriov_configure to fix build errors w/o SR-IOV in kernel
> v4: Dropped vfio-pci patch
> Added ena and nvme to drivers now using pci_sriov_configure_unmanaged
> Dropped pci_disable_sriov call in virtio_pci to be consistent with ena
> v5: Dropped sriov_unmanaged_autoprobe and pci_sriov_conifgure_unmanaged
> Added new patch that enables pci_sriov_configure_simple
> Updated drivers to use pci_sriov_configure_simple
> v6: Defined pci_sriov_configure_simple as NULL when SR-IOV is not enabled
> Updated drivers to drop "#ifdef" checks for IOV
> Added pci-pf-stub as place for PF-only drivers to add support
> v7: Dropped pci_id table explanation from pci-pf-stub driver
> Updated pci_sriov_configure_simple to drop need for err value
> Fixed comment explaining why pci_sriov_configure_simple is NULL
> v8: Dropped virtio from the set, support to be added later after TC approval
>
> Cc: Mark Rustad <mark.d.rustad@intel.com>
> Cc: Maximilian Heyne <mheyne@amazon.de>
> Cc: Liang-Min Wang <liang-min.wang@intel.com>
> Cc: David Woodhouse <dwmw@amazon.co.uk>
>
> ---
>
> Alexander Duyck (4):
> pci: Add pci_sriov_configure_simple for PFs that don't manage VF resources
> ena: Migrate over to unmanaged SR-IOV support
> nvme: Migrate over to unmanaged SR-IOV support
> pci-pf-stub: Add PF driver stub for PFs that function only to enable VFs
>
>
> drivers/net/ethernet/amazon/ena/ena_netdev.c | 28 -------------
> drivers/nvme/host/pci.c | 20 ----------
> drivers/pci/Kconfig | 12 ++++++
> drivers/pci/Makefile | 2 +
> drivers/pci/iov.c | 31 +++++++++++++++
> drivers/pci/pci-pf-stub.c | 54 ++++++++++++++++++++++++++
> include/linux/pci.h | 3 +
> include/linux/pci_ids.h | 2 +
> 8 files changed, 106 insertions(+), 46 deletions(-)
> create mode 100644 drivers/pci/pci-pf-stub.c
>
> --
>
--
~Randy
^ permalink raw reply
* Re: [PATCH] PCI: Add PCIe to pcie_print_link_status() messages
From: Bjorn Helgaas @ 2018-04-20 17:38 UTC (permalink / raw)
To: Jakub Kicinski
Cc: oss-drivers, Tal Gilboa, Tariq Toukan, Jacob Keller,
Ganesh Goudar, Jeff Kirsher, intel-wired-lan, netdev,
linux-kernel, linux-pci
In-Reply-To: <20180413181638.6424-1-jakub.kicinski@netronome.com>
On Fri, Apr 13, 2018 at 11:16:38AM -0700, Jakub Kicinski wrote:
> Currently the pcie_print_link_status() will print PCIe bandwidth
> and link width information but does not mention it is pertaining
> to the PCIe. Since this and related functions are used exclusively
> by networking drivers today users may get confused into thinking
> that it's the NIC bandwidth that is being talked about. Insert a
> "PCIe" into the messages.
>
> Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Applied to for-linus for v4.17, thanks!
> ---
> drivers/pci/pci.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index aa86e904f93c..73a0a4993f6a 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -5273,11 +5273,11 @@ void pcie_print_link_status(struct pci_dev *dev)
> bw_avail = pcie_bandwidth_available(dev, &limiting_dev, &speed, &width);
>
> if (bw_avail >= bw_cap)
> - pci_info(dev, "%u.%03u Gb/s available bandwidth (%s x%d link)\n",
> + pci_info(dev, "%u.%03u Gb/s available PCIe bandwidth (%s x%d link)\n",
> bw_cap / 1000, bw_cap % 1000,
> PCIE_SPEED2STR(speed_cap), width_cap);
> else
> - pci_info(dev, "%u.%03u Gb/s available bandwidth, limited by %s x%d link at %s (capable of %u.%03u Gb/s with %s x%d link)\n",
> + pci_info(dev, "%u.%03u Gb/s available PCIe bandwidth, limited by %s x%d link at %s (capable of %u.%03u Gb/s with %s x%d link)\n",
> bw_avail / 1000, bw_avail % 1000,
> PCIE_SPEED2STR(speed), width,
> limiting_dev ? pci_name(limiting_dev) : "<unknown>",
> --
> 2.16.2
>
^ permalink raw reply
* Re: [PATCH RFC net-next 00/11] udp gso
From: Alexander Duyck @ 2018-04-20 17:38 UTC (permalink / raw)
To: Willem de Bruijn
Cc: David Miller, Sowmini Varadhan, Samudrala, Sridhar, Netdev,
Willem de Bruijn
In-Reply-To: <CAF=yD-+V7B67NbL8aELa9QR0Hx8bAvKNQ=JfjkwGGFtey-2FOw@mail.gmail.com>
On Wed, Apr 18, 2018 at 11:22 AM, Willem de Bruijn
<willemdebruijn.kernel@gmail.com> wrote:
> On Wed, Apr 18, 2018 at 2:12 PM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>> On Wed, Apr 18, 2018 at 10:28 AM, David Miller <davem@davemloft.net> wrote:
>>> From: Sowmini Varadhan <sowmini.varadhan@oracle.com>
>>> Date: Wed, 18 Apr 2018 08:31:03 -0400
>>>
>>>> However, I share Sridhar's concerns about the very fundamental change
>>>> to UDP message boundary semantics here. There is actually no such thing
>>>> as a "segment" in udp, so in general this feature makes me a little
>>>> uneasy. Well behaved udp applications should already be sending mtu
>>>> sized datagrams. And the not-so-well-behaved ones are probably relying
>>>> on IP fragmentation/reassembly to take care of datagram boundary semantics
>>>> for them?
>>>>
>>>> As Sridhar points out, the feature is not really "negotiated" - one side
>>>> unilaterally sets the option. If the receiver is a classic/POSIX UDP
>>>> implementation, it will have no way of knowing that message boundaries
>>>> have been re-adjusted at the sender.
>>>
>>> There are no "semantics".
>>>
>>> What ends up on the wire is the same before the kernel/app changes as
>>> afterwards.
>>>
>>> The only difference is that instead of the application doing N - 1
>>> sendmsg() calls with mtu sized writes, it's giving everything all at
>>> once and asking the kernel to segment.
>>>
>>> It even gives the application control over the size of the packets,
>>> which I think is completely prudent in this situation.
>>
>> My only concern with the patch set is verifying what mitigations are
>> in case so that we aren't trying to set an MSS size that results in a
>> frame larger than MTU. I'm still digging through the code and trying
>> to grok it, but I figured I might just put the question out there to
>> may my reviewing easier.
>
> This is checked in udp_send_skb in
>
> const int hlen = skb_network_header_len(skb) +
> sizeof(struct udphdr);
>
> if (hlen + cork->gso_size > cork->fragsize)
> return -EINVAL;
>
> At this point cork->fragsize will have been set in ip_setup_cork
> based on device or path mtu:
>
> cork->fragsize = ip_sk_use_pmtu(sk) ?
> dst_mtu(&rt->dst) : rt->dst.dev->mtu;
>
> The feature bypasses the MTU sanity checks in __ip_append_data
> by setting local variable mtu to a network layer max
>
> mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize;
>
> but the above should perform the same check, only later. The
> check in udp_send_skb is simpler as it does not have to deal
> with the case of fragmentation.
>
>> Also any plans for HW offload support for this? I vaguely recall that
>> the igb and ixgbe parts had support for something like this in
>> hardware. I would have to double check to see what exactly is
>> supported.
>
> I hadn't given that much thought until the request yesterday to
> expose the NETIF_F_GSO_UDP_L4 flag through ethtool. By
> virtue of having only a single fixed segmentation length, it
> appears reasonably straightforward to offload.
Actually I just got a chance to start on a review of things. Do we
need to have to use both GSO_UDP and and GSO_UDP_L4? It might be
better if we could split these up and specifically call out GSO_UDP as
UFO and GSO_UDP_L4 as being UDP segmentation.
My only concern is that I don't think devices would want to have to
try and advertise both GSO_UDP and GSO_UDP_L4 if they want to support
UDP segmentation only. Ideally we want to keep UFO separate from UDP
segmentation since they would be two different things.
Thanks.
- Alex
^ permalink raw reply
* Re: [PATCH RFC net-next 00/11] udp gso
From: Tushar Dave @ 2018-04-20 18:27 UTC (permalink / raw)
To: Alexander Duyck, David Miller
Cc: Sowmini Varadhan, Willem de Bruijn, Samudrala, Sridhar, Netdev,
Willem de Bruijn
In-Reply-To: <CAKgT0UcefCpG2j7RQN8aUUgu2bud_RTFs_s+nFPY2GhKgE8L+A@mail.gmail.com>
On 04/18/2018 11:12 AM, Alexander Duyck wrote:
> On Wed, Apr 18, 2018 at 10:28 AM, David Miller <davem@davemloft.net> wrote:
>> From: Sowmini Varadhan <sowmini.varadhan@oracle.com>
>> Date: Wed, 18 Apr 2018 08:31:03 -0400
>>
>>> However, I share Sridhar's concerns about the very fundamental change
>>> to UDP message boundary semantics here. There is actually no such thing
>>> as a "segment" in udp, so in general this feature makes me a little
>>> uneasy. Well behaved udp applications should already be sending mtu
>>> sized datagrams. And the not-so-well-behaved ones are probably relying
>>> on IP fragmentation/reassembly to take care of datagram boundary semantics
>>> for them?
>>>
>>> As Sridhar points out, the feature is not really "negotiated" - one side
>>> unilaterally sets the option. If the receiver is a classic/POSIX UDP
>>> implementation, it will have no way of knowing that message boundaries
>>> have been re-adjusted at the sender.
>>
>> There are no "semantics".
>>
>> What ends up on the wire is the same before the kernel/app changes as
>> afterwards.
>>
>> The only difference is that instead of the application doing N - 1
>> sendmsg() calls with mtu sized writes, it's giving everything all at
>> once and asking the kernel to segment.
>>
>> It even gives the application control over the size of the packets,
>> which I think is completely prudent in this situation.
>
> My only concern with the patch set is verifying what mitigations are
> in case so that we aren't trying to set an MSS size that results in a
> frame larger than MTU. I'm still digging through the code and trying
> to grok it, but I figured I might just put the question out there to
> may my reviewing easier.
>
> Also any plans for HW offload support for this? I vaguely recall that
> the igb and ixgbe parts had support for something like this in
> hardware. I would have to double check to see what exactly is
> supported.
Alex,
If by HW support you meant UFO (UDP Fragmentation Offload), then I have
dig into that last year using ixgbe. And I found that Intel 10G HW does
break large UDP packets into MTU size however it does not generate
*true* IP fragments. Instead, when large (> MTU) size UDP packet is
given to NIC, HW generates unique UDP packets with distinct IP
fragments. This makes it impossible for receiving station to reassemble
them into one UDP packet.
I am not sure about igb!
-Tushar
>
> Thanks.
>
> - Alex
>
^ permalink raw reply
* Re: [PATCH net-next 0/5] virtio-net: Add SCTP checksum offload support
From: Michael S. Tsirkin @ 2018-04-20 18:32 UTC (permalink / raw)
To: Marcelo Ricardo Leitner
Cc: Vlad Yasevich, Vladislav Yasevich, netdev, linux-sctp,
virtualization, jasowang, nhorman
In-Reply-To: <20180420172219.GR4716@localhost.localdomain>
On Fri, Apr 20, 2018 at 02:22:19PM -0300, Marcelo Ricardo Leitner wrote:
> On Wed, Apr 18, 2018 at 05:06:46PM +0300, Michael S. Tsirkin wrote:
> > On Tue, Apr 17, 2018 at 04:35:18PM -0400, Vlad Yasevich wrote:
> > > On 04/02/2018 10:47 AM, Marcelo Ricardo Leitner wrote:
> > > > On Mon, Apr 02, 2018 at 09:40:01AM -0400, Vladislav Yasevich wrote:
> > > >> Now that we have SCTP offload capabilities in the kernel, we can add
> > > >> them to virtio as well. First step is SCTP checksum.
> > > >
> > > > Thanks.
> > > >
> > > >> As for GSO, the way sctp GSO is currently implemented buys us nothing
> > > >> in added support to virtio. To add true GSO, would require a lot of
> > > >> re-work inside of SCTP and would require extensions to the virtio
> > > >> net header to carry extra sctp data.
> > > >
> > > > Can you please elaborate more on this? Is this because SCTP GSO relies
> > > > on the gso skb format for knowing how to segment it instead of having
> > > > a list of sizes?
> > > >
> > >
> > > it's mainly because all the true segmentation, placing data into chunks,
> > > has already happened. All that GSO does is allow for higher bundling
> > > rate between VMs. If that is all SCTP GSO ever going to do, that fine,
> > > but the goal is to do real GSO eventually and potentially reduce the
> > > amount of memory copying we are doing.
> > > If we do that, any current attempt at GSO in virtio would have to be
> > > depricated and we'd need GSO2 or something like that.
> >
> > Batching helps virtualization *a lot* though.
>
> Yep. The results posted by Xin in the other email give good insights
> on it.
>
> > Are there actual plans for GSO2? Is it just for SCTP?
>
> No plans. In this context, at least, yes, just for SCTP.
>
> It was a supposition in case we start doing a different GSO for SCTP,
> one more like what we have for TCP.
>
> Currently, as the SCTP GSO code doesn't leave the system, we can
> update it if we want. But by the moment we add support for it in
> virtio, we will have to be backwards compatible if we end up doing
> SCTP GSO differently.
At least for TX you can always just disable the optimization. Won't be
worse than what is there now. RX is trickier - but that's GRO
not GSO.
> But again, I don't think such approach for SCTP GSO would be neither
> feasible or worth. The complexity for it, to work across stream
> schedules and late TSN allocation, would do more harm then good IMO.
>
> >
> > >
> > > This is why, after doing the GSO support, I decided not to include it.
> > >
> > > -vlad
> > > > Marcelo
> > > >
^ permalink raw reply
* Re: [PATCH bpf-next v3 4/8] bpf: add documentation for eBPF helpers (23-32)
From: Quentin Monnet @ 2018-04-20 18:54 UTC (permalink / raw)
To: Daniel Borkmann, ast; +Cc: netdev, oss-drivers, linux-doc, linux-man
In-Reply-To: <6f1b43c7-5d79-7419-1053-d0b29c1e2bb9@iogearbox.net>
2018-04-19 13:16 UTC+0200 ~ Daniel Borkmann <daniel@iogearbox.net>
> On 04/17/2018 04:34 PM, Quentin Monnet wrote:
>> Add documentation for eBPF helper functions to bpf.h user header file.
>> This documentation can be parsed with the Python script provided in
>> another commit of the patch series, in order to provide a RST document
>> that can later be converted into a man page.
>>
>> The objective is to make the documentation easily understandable and
>> accessible to all eBPF developers, including beginners.
>>
>> This patch contains descriptions for the following helper functions, all
>> written by Daniel:
>>
>> - bpf_get_prandom_u32()
>> - bpf_get_smp_processor_id()
>> - bpf_get_cgroup_classid()
>> - bpf_get_route_realm()
>> - bpf_skb_load_bytes()
>> - bpf_csum_diff()
>> - bpf_skb_get_tunnel_opt()
>> - bpf_skb_set_tunnel_opt()
>> - bpf_skb_change_proto()
>> - bpf_skb_change_type()
>>
>> v3:
>> - bpf_get_prandom_u32(): Fix helper name :(. Add description, including
>> a note on the internal random state.
>> - bpf_get_smp_processor_id(): Add description, including a note on the
>> processor id remaining stable during program run.
>> - bpf_get_cgroup_classid(): State that CONFIG_CGROUP_NET_CLASSID is
>> required to use the helper. Add a reference to related documentation.
>> State that placing a task in net_cls controller disables cgroup-bpf.
>> - bpf_get_route_realm(): State that CONFIG_CGROUP_NET_CLASSID is
>> required to use this helper.
>> - bpf_skb_load_bytes(): Fix comment on current use cases for the helper.
>>
>> Cc: Daniel Borkmann <daniel@iogearbox.net>
>> Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com>
>> ---
>> include/uapi/linux/bpf.h | 152 +++++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 152 insertions(+)
>>
>> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
>> index c59bf5b28164..d748f65a8f58 100644
>> --- a/include/uapi/linux/bpf.h
>> +++ b/include/uapi/linux/bpf.h
[...]
>> @@ -615,6 +632,27 @@ union bpf_attr {
>> * Return
>> * 0 on success, or a negative error in case of failure.
>> *
>> + * u32 bpf_get_cgroup_classid(struct sk_buff *skb)
>> + * Description
>> + * Retrieve the classid for the current task, i.e. for the
>> + * net_cls (network classifier) cgroup to which *skb* belongs.
>> + *
>> + * This helper is only available is the kernel was compiled with
>> + * the **CONFIG_CGROUP_NET_CLASSID** configuration option set to
>> + * "**y**" or to "**m**".
>> + *
>> + * Note that placing a task into the net_cls controller completely
>> + * disables the execution of eBPF programs with the cgroup.
>
> I'm not sure I follow the above sentence, what do you mean by that?
Please see comment below.
> I would definitely also add here that this helper is limited to cgroups v1
> only, and that it works on clsact TC egress hook but not the ingress one.
>
>> + * Also note that, in the above description, the "network
>> + * classifier" cgroup does not designate a generic classifier, but
>> + * a particular mechanism that provides an interface to tag
>> + * network packets with a specific class identifier. See also the
>
> The "generic classifier" part is a bit strange to parse. I would probably
> leave the first part out and explain that this provides a means to tag
> packets based on a user-provided ID for all traffic coming from the tasks
> belonging to the related cgroup.
In this paragraph and in the one above, I am trying to address Alexei
comments to the RFC v2:
please add that kernel should be configured with
CONFIG_NET_CLS_CGROUP=y|m
and mention Documentation/cgroup-v1/net_cls.txt
Otherwise 'network classifier' is way too generic.
I'd also mention that placing a task into net_cls controller
disables all of cgroup-bpf.
But I must admit I am struggling a bit on this helper. If I understand
correctly, "network classifier" is too generic and could be confused
with TC classifiers? Hence the attempt to avoid that in the second
paragraph... As for "placing a task into net_cls controller disables all
of cgroup-bpf", again if I understand correctly, I think it refers to
cgroup_sk_alloc_disable() being called in write_classid() in file
net/core/netclassid_cgroup.c. But I am not familiar with it and cannot
find a nice way to explain that in the doc :/.
>> + * related kernel documentation, available from the Linux sources
>> + * in file *Documentation/cgroup-v1/net_cls.txt*.
>> + * Return
>> + * The classid, or 0 for the default unconfigured classid.
>> + *
>> * int bpf_skb_vlan_push(struct sk_buff *skb, __be16 vlan_proto, u16 vlan_tci)
>> * Description
>> * Push a *vlan_tci* (VLAN tag control information) of protocol
>> @@ -734,6 +772,16 @@ union bpf_attr {
>> * are **TC_ACT_REDIRECT** on success or **TC_ACT_SHOT** on
>> * error.
>> *
>> + * u32 bpf_get_route_realm(struct sk_buff *skb)
>> + * Description
>> + * Retrieve the realm or the route, that is to say the
>> + * **tclassid** field of the destination for the *skb*. This
>> + * helper is available only if the kernel was compiled with
>> + * **CONFIG_IP_ROUTE_CLASSID** configuration option.
>
> Could mention that this is a similar user provided tag like in the net_cls
> case with cgroups only that this applies to routes here (dst entries) which
> hold this tag.
>
> Also, should say that this works with clsact TC egress hook or alternatively
> conventional classful egress qdiscs, but not on TC ingress. In case of clsact
> TC egress hook this has the advantage that the dst entry has not been dropped
> yet in the xmit path. Therefore, the dst entry does not need to be artificially
> held via netif_keep_dst() for a classful qdisc until the skb is freed.
Here I am not sure to follow, the advantage is for clsact, but this is
only for a classful qdisc that we can avoid holding the dst entry? How
does holding the entry with netif_keep_dst() translate, from a user
space point of view?
Thanks a lot for the exhaustive review!
Quentin
^ permalink raw reply
* [PATCH net] strparser: Do not call mod_delayed_work with a timeout of LONG_MAX
From: Doron Roberts-Kedes @ 2018-04-20 19:11 UTC (permalink / raw)
To: David Miller; +Cc: Tejun Heo, netdev, Doron Roberts-Kedes
struct sock's sk_rcvtimeo is initialized to
LONG_MAX/MAX_SCHEDULE_TIMEOUT in sock_init_data. Calling
mod_delayed_work with a timeout of LONG_MAX causes spurious execution of
the work function. timer->expires is set equal to jiffies + LONG_MAX.
When timer_base->clk falls behind the current value of jiffies,
the delta between timer_base->clk and jiffies + LONG_MAX causes the
expiration to be in the past. Returning early from strp_start_timer if
timeo == LONG_MAX solves this problem.
Found while testing net/tls_sw recv path.
Fixes: 43a0c6751a322847 ("strparser: Stream parser for messages")
Reviewed-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Doron Roberts-Kedes <doronrk@fb.com>
---
net/strparser/strparser.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/net/strparser/strparser.c b/net/strparser/strparser.c
index 805b139..092bebc 100644
--- a/net/strparser/strparser.c
+++ b/net/strparser/strparser.c
@@ -67,7 +67,7 @@ static void strp_abort_strp(struct strparser *strp, int err)
static void strp_start_timer(struct strparser *strp, long timeo)
{
- if (timeo)
+ if (timeo && timeo != LONG_MAX)
mod_delayed_work(strp_wq, &strp->msg_timer_work, timeo);
}
--
2.9.5
^ permalink raw reply related
* [PATCHv4 net 0/3] net: sched: ife: malformed ife packet fixes
From: Alexander Aring @ 2018-04-20 19:15 UTC (permalink / raw)
To: yotam.gi
Cc: jhs, davem, xiyou.wangcong, jiri, yuvalm, netdev, kernel,
Alexander Aring
As promised at netdev 2.2 tc workshop I am working on adding scapy support for
tdc testing. It is still work in progress. I will submit the patches to tdc
later (they are not in good shape yet). The good news is I have been able to
find bugs which normal packet testing would not be able to find.
With fuzzy testing I was able to craft certain malformed packets that IFE
action was not able to deal with. This patch set fixes those bugs.
changes since v4:
- use pskb_may_pull before pointer assign
changes since v3:
- use pskb_may_pull
changes since v2:
- remove inline from __ife_tlv_meta_valid
- add const to cast to meta_tlvhdr
- add acked and reviewed tags
Alexander Aring (3):
net: sched: ife: signal not finding metaid
net: sched: ife: handle malformed tlv length
net: sched: ife: check on metadata length
include/net/ife.h | 3 ++-
net/ife/ife.c | 38 ++++++++++++++++++++++++++++++++++++--
net/sched/act_ife.c | 9 +++++++--
3 files changed, 45 insertions(+), 5 deletions(-)
--
2.11.0
^ permalink raw reply
* [PATCHv4 net 1/3] net: sched: ife: signal not finding metaid
From: Alexander Aring @ 2018-04-20 19:15 UTC (permalink / raw)
To: yotam.gi
Cc: jhs, davem, xiyou.wangcong, jiri, yuvalm, netdev, kernel,
Alexander Aring
In-Reply-To: <20180420191505.27633-1-aring@mojatatu.com>
We need to record stats for received metadata that we dont know how
to process. Have find_decode_metaid() return -ENOENT to capture this.
Signed-off-by: Alexander Aring <aring@mojatatu.com>
Reviewed-by: Yotam Gigi <yotam.gi@gmail.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
---
net/sched/act_ife.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
index a5994cf0512b..49b8ab551fbe 100644
--- a/net/sched/act_ife.c
+++ b/net/sched/act_ife.c
@@ -652,7 +652,7 @@ static int find_decode_metaid(struct sk_buff *skb, struct tcf_ife_info *ife,
}
}
- return 0;
+ return -ENOENT;
}
static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a,
--
2.11.0
^ permalink raw reply related
* [PATCHv4 net 2/3] net: sched: ife: handle malformed tlv length
From: Alexander Aring @ 2018-04-20 19:15 UTC (permalink / raw)
To: yotam.gi
Cc: jhs, davem, xiyou.wangcong, jiri, yuvalm, netdev, kernel,
Alexander Aring
In-Reply-To: <20180420191505.27633-1-aring@mojatatu.com>
There is currently no handling to check on a invalid tlv length. This
patch adds such handling to avoid killing the kernel with a malformed
ife packet.
Signed-off-by: Alexander Aring <aring@mojatatu.com>
Reviewed-by: Yotam Gigi <yotam.gi@gmail.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
---
include/net/ife.h | 3 ++-
net/ife/ife.c | 35 +++++++++++++++++++++++++++++++++--
net/sched/act_ife.c | 7 ++++++-
3 files changed, 41 insertions(+), 4 deletions(-)
diff --git a/include/net/ife.h b/include/net/ife.h
index 44b9c00f7223..e117617e3c34 100644
--- a/include/net/ife.h
+++ b/include/net/ife.h
@@ -12,7 +12,8 @@
void *ife_encode(struct sk_buff *skb, u16 metalen);
void *ife_decode(struct sk_buff *skb, u16 *metalen);
-void *ife_tlv_meta_decode(void *skbdata, u16 *attrtype, u16 *dlen, u16 *totlen);
+void *ife_tlv_meta_decode(void *skbdata, const void *ifehdr_end, u16 *attrtype,
+ u16 *dlen, u16 *totlen);
int ife_tlv_meta_encode(void *skbdata, u16 attrtype, u16 dlen,
const void *dval);
diff --git a/net/ife/ife.c b/net/ife/ife.c
index 7d1ec76e7f43..7fbe70a0af4b 100644
--- a/net/ife/ife.c
+++ b/net/ife/ife.c
@@ -92,12 +92,43 @@ struct meta_tlvhdr {
__be16 len;
};
+static bool __ife_tlv_meta_valid(const unsigned char *skbdata,
+ const unsigned char *ifehdr_end)
+{
+ const struct meta_tlvhdr *tlv;
+ u16 tlvlen;
+
+ if (unlikely(skbdata + sizeof(*tlv) > ifehdr_end))
+ return false;
+
+ tlv = (const struct meta_tlvhdr *)skbdata;
+ tlvlen = ntohs(tlv->len);
+
+ /* tlv length field is inc header, check on minimum */
+ if (tlvlen < NLA_HDRLEN)
+ return false;
+
+ /* overflow by NLA_ALIGN check */
+ if (NLA_ALIGN(tlvlen) < tlvlen)
+ return false;
+
+ if (unlikely(skbdata + NLA_ALIGN(tlvlen) > ifehdr_end))
+ return false;
+
+ return true;
+}
+
/* Caller takes care of presenting data in network order
*/
-void *ife_tlv_meta_decode(void *skbdata, u16 *attrtype, u16 *dlen, u16 *totlen)
+void *ife_tlv_meta_decode(void *skbdata, const void *ifehdr_end, u16 *attrtype,
+ u16 *dlen, u16 *totlen)
{
- struct meta_tlvhdr *tlv = (struct meta_tlvhdr *) skbdata;
+ struct meta_tlvhdr *tlv;
+
+ if (!__ife_tlv_meta_valid(skbdata, ifehdr_end))
+ return NULL;
+ tlv = (struct meta_tlvhdr *)skbdata;
*dlen = ntohs(tlv->len) - NLA_HDRLEN;
*attrtype = ntohs(tlv->type);
diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
index 49b8ab551fbe..8527cfdc446d 100644
--- a/net/sched/act_ife.c
+++ b/net/sched/act_ife.c
@@ -682,7 +682,12 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a,
u16 mtype;
u16 dlen;
- curr_data = ife_tlv_meta_decode(tlv_data, &mtype, &dlen, NULL);
+ curr_data = ife_tlv_meta_decode(tlv_data, ifehdr_end, &mtype,
+ &dlen, NULL);
+ if (!curr_data) {
+ qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats));
+ return TC_ACT_SHOT;
+ }
if (find_decode_metaid(skb, ife, mtype, dlen, curr_data)) {
/* abuse overlimits to count when we receive metadata
--
2.11.0
^ permalink raw reply related
* [PATCHv4 net 3/3] net: sched: ife: check on metadata length
From: Alexander Aring @ 2018-04-20 19:15 UTC (permalink / raw)
To: yotam.gi
Cc: jhs, davem, xiyou.wangcong, jiri, yuvalm, netdev, kernel,
Alexander Aring
In-Reply-To: <20180420191505.27633-1-aring@mojatatu.com>
This patch checks if sk buffer is available to dererence ife header. If
not then NULL will returned to signal an malformed ife packet. This
avoids to crashing the kernel from outside.
Signed-off-by: Alexander Aring <aring@mojatatu.com>
Reviewed-by: Yotam Gigi <yotam.gi@gmail.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
---
net/ife/ife.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/net/ife/ife.c b/net/ife/ife.c
index 7fbe70a0af4b..13bbf8cb6a39 100644
--- a/net/ife/ife.c
+++ b/net/ife/ife.c
@@ -69,6 +69,9 @@ void *ife_decode(struct sk_buff *skb, u16 *metalen)
int total_pull;
u16 ifehdrln;
+ if (!pskb_may_pull(skb, skb->dev->hard_header_len + IFE_METAHDRLEN))
+ return NULL;
+
ifehdr = (struct ifeheadr *) (skb->data + skb->dev->hard_header_len);
ifehdrln = ntohs(ifehdr->metalen);
total_pull = skb->dev->hard_header_len + ifehdrln;
--
2.11.0
^ permalink raw reply related
* [PATCH net] ibmvnic: Clean actual number of RX or TX pools
From: Thomas Falcon @ 2018-04-20 19:25 UTC (permalink / raw)
To: netdev; +Cc: nfont, jallen, linuxppc-dev, Thomas Falcon
Avoid using value stored in the login response buffer when
cleaning TX and RX buffer pools since these could be inconsistent
depending on the device state. Instead use the field in the driver's
private data that tracks the number of active pools.
Signed-off-by: Thomas Falcon <tlfalcon@linux.vnet.ibm.com>
---
drivers/net/ethernet/ibm/ibmvnic.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 2df01ad..6e8d6a6 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -1128,7 +1128,7 @@ static void clean_rx_pools(struct ibmvnic_adapter *adapter)
if (!adapter->rx_pool)
return;
- rx_scrqs = be32_to_cpu(adapter->login_rsp_buf->num_rxadd_subcrqs);
+ rx_scrqs = adapter->num_active_rx_pools;
rx_entries = adapter->req_rx_add_entries_per_subcrq;
/* Free any remaining skbs in the rx buffer pools */
@@ -1177,7 +1177,7 @@ static void clean_tx_pools(struct ibmvnic_adapter *adapter)
if (!adapter->tx_pool || !adapter->tso_pool)
return;
- tx_scrqs = be32_to_cpu(adapter->login_rsp_buf->num_txsubm_subcrqs);
+ tx_scrqs = adapter->num_active_tx_pools;
/* Free any remaining skbs in the tx buffer pools */
for (i = 0; i < tx_scrqs; i++) {
--
1.8.3.1
^ permalink raw reply related
* [PATCH RFC net-next] igb: adjust SYSTIM register using TIMADJ register
From: Kshitiz Gupta @ 2018-04-20 19:56 UTC (permalink / raw)
To: jeffrey.t.kirsher, richardcochran
Cc: intel-wired-lan, netdev, linux-kernel, Kshitiz Gupta
Currently the driver adjusts time by reading the current time and then
modifying it before writing to SYSTIM register. This can introduce
inaccuracies in SYSTIM. With a PREEMPT_RT kernel, spinlocks may be
interrupted, which in the existing implementation may lead to increased
time between the read and the write.
Alternatively as per section 7.8.3.2 in the i210 data sheet, this
operation can be done more accurately by using the TIMADJ registers,
but this should only be used for adjustments less than one 8th of the
sync interval. Once this register is written, the software can poll on
TSICR.TADJ to make sure that adjustment operation is completed.
This change implements using TIMADJ registers for adjTime call for
deltas less a configurable threshold. This threshold is exposed as
configurable module parameter. Once the delta is written to the
register, driver polls on TSICR.TADJ register to make sure the
adjustment is finished. For deltas greater than this threshold the
driver reverts back to using the old Read-Modify-Write approach.
Signed-off-by: Kshitiz Gupta <kshitiz.gupta@ni.com>
---
This change does create an oddity in the way the adjument takes place.
For any adjustments greater than the threshold the operation is
"immediate", but for the case where this new approach of TIMADJ is
used it can lead to the driver blocking for 8ns * delta.
There is another approach that could be taken. The driver could write
the TIMADJ and return immediately. Any subsequent could block until the
operation is done or return EINPROGRESS to let the caller know to
retry.
---
drivers/net/ethernet/intel/igb/e1000_regs.h | 1 +
drivers/net/ethernet/intel/igb/igb_ptp.c | 64 ++++++++++++++++++++++++++---
2 files changed, 59 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/intel/igb/e1000_regs.h b/drivers/net/ethernet/intel/igb/e1000_regs.h
index d84afdd..607e0ee 100644
--- a/drivers/net/ethernet/intel/igb/e1000_regs.h
+++ b/drivers/net/ethernet/intel/igb/e1000_regs.h
@@ -100,6 +100,7 @@
#define E1000_SYSTIML 0x0B600 /* System time register Low - RO */
#define E1000_SYSTIMH 0x0B604 /* System time register High - RO */
#define E1000_TIMINCA 0x0B608 /* Increment attributes register - RW */
+#define E1000_TIMADJ 0x0B60C /* Time Adjustment Offset Register - RW */
#define E1000_TSAUXC 0x0B640 /* Timesync Auxiliary Control register */
#define E1000_TRGTTIML0 0x0B644 /* Target Time Register 0 Low - RW */
#define E1000_TRGTTIMH0 0x0B648 /* Target Time Register 0 High - RW */
diff --git a/drivers/net/ethernet/intel/igb/igb_ptp.c b/drivers/net/ethernet/intel/igb/igb_ptp.c
index 9714b7f..6e9e0ac 100644
--- a/drivers/net/ethernet/intel/igb/igb_ptp.c
+++ b/drivers/net/ethernet/intel/igb/igb_ptp.c
@@ -76,6 +76,16 @@
static void igb_ptp_tx_hwtstamp(struct igb_adapter *adapter);
+/* If adjTime is called with delta less that timadj_threshold the driver uses
+ * the TIMADJ register to update the SYSTIM register. This is used only for
+ * deltas which are less than 1/8th of the SYNC interval. Using default value
+ * of 15.6ms, which happens to be 1/8th of the default SYNC interval for
+ * IEEE 802.1AS-2011.
+ */
+static int timadj_threshold = 15625000;
+module_param(timadj_threshold, int, 0644);
+MODULE_PARM_DESC(timadj_threshold, "Threshold under which TIMADJ will be used to update SYSTIM");
+
/* SYSTIM read access for the 82576 */
static cycle_t igb_ptp_read_82576(const struct cyclecounter *cc)
{
@@ -269,18 +279,60 @@ static int igb_ptp_adjtime_i210(struct ptp_clock_info *ptp, s64 delta)
{
struct igb_adapter *igb = container_of(ptp, struct igb_adapter,
ptp_caps);
- unsigned long flags;
+ unsigned long flags, timeout_us, i, wait_us = 1000;
+ unsigned long clock_interval_ns = 8;
struct timespec64 now, then = ns_to_timespec64(delta);
+ struct e1000_hw *hw = &igb->hw;
+ u32 adjust_offset = 0, tsicr = 0;
+ int ret = 0;
spin_lock_irqsave(&igb->tmreg_lock, flags);
- igb_ptp_read_i210(igb, &now);
- now = timespec64_add(now, then);
- igb_ptp_write_i210(igb, (const struct timespec64 *)&now);
+ if (abs(delta) > timadj_threshold) {
+ igb_ptp_read_i210(igb, &now);
+ now = timespec64_add(now, then);
+ igb_ptp_write_i210(igb, (const struct timespec64 *)&now);
+ } else {
+ /* For smaller values based on the threshold set in the module
+ * parameter adjust SYSTIM registers using TIMADJ registers.
+ * Following the write access to the TIMADJ register, the
+ * hardware at every 8ns clock repeats the following two steps:
+ *
+ * SYSTIM = SYSTIM + INC_TIME +/- 1 nsec. Add or subtract 1 nsec
+ * is defined by TIMADJ.Sign ( 0 means Add and 1 means Subtract)
+ *
+ * Tadjust = Tadjust - 1 nsec
+ */
- spin_unlock_irqrestore(&igb->tmreg_lock, flags);
+ /* TIMADJ is one's complement, set the magnitude and sign */
+ adjust_offset = abs(delta);
+ if (delta < 0)
+ adjust_offset |= ISGN;
- return 0;
+ wr32(E1000_TIMADJ, adjust_offset);
+
+ /* Poll on TSICR.TADJ flag. This is set when hardware completes
+ * the adjustment based on TIMADJ register.
+ */
+ timeout_us = timadj_threshold * clock_interval_ns;
+ for (i = 0; i < timeout_us; i += wait_us) {
+ tsicr = rd32(E1000_TSICR);
+
+ if (tsicr & TSINTR_TADJ)
+ break;
+
+ udelay(wait_us);
+ }
+
+ if (i >= timeout_us) {
+ tsicr = rd32(E1000_TSICR);
+ if (!(tsicr & TSINTR_TADJ))
+ ret = -ETIMEDOUT;
+ }
+ }
+
+ spin_unlock_irqrestore(&igb->tmreg_lock, flags);
+ return ret;
}
static int igb_ptp_gettime_82576(struct ptp_clock_info *ptp,
--
1.9.1
^ permalink raw reply related
* Re: [pci PATCH v8 0/4] Add support for unmanaged SR-IOV
From: Alexander Duyck @ 2018-04-20 20:01 UTC (permalink / raw)
To: Randy Dunlap
Cc: Alexander Duyck, Bjorn Helgaas, linux-pci, virtio-dev, kvm,
Netdev, Daly, Dan, LKML, linux-nvme, Keith Busch, netanel,
Don Dutile, Maximilian Heyne, Wang, Liang-min, Rustad, Mark D,
David Woodhouse, Christoph Hellwig, dwmw
In-Reply-To: <5fc62d10-790f-2356-c2f6-fd49f542dd3c@infradead.org>
On Fri, Apr 20, 2018 at 10:23 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
> On 04/20/18 09:28, Alexander Duyck wrote:
>> This series is meant to add support for SR-IOV on devices when the VFs are
>> not managed by the kernel. Examples of recent patches attempting to do this
>> include:
>> virto - https://patchwork.kernel.org/patch/10241225/
>> pci-stub - https://patchwork.kernel.org/patch/10109935/
>> vfio - https://patchwork.kernel.org/patch/10103353/
>> uio - https://patchwork.kernel.org/patch/9974031/
>
> Hi,
>
> Somewhere in this patch series it would be nice to tell us what the heck
> a "PF" is. :)
>
> Thanks.
Sorry, I was kind of operating on the assumption of everyone
understanding SR-IOV nomenclature.
A "PF" is a PCIe Physical Function. When you bring up a PCIe device
that supports SR-IOV it is the device that is there to begin with.
A "VF" is a PCIe Virtual Function. You could think of as a logical
device that is spawned from the physical function using a combination
of hardware configuration via the SR-IOV block in the PCIe extended
configuration space and kernel/driver features.
There are also a number of online resources you could use to research
SR-IOV further. Hope that helps to clarify some of this.
Thanks.
Alex
^ permalink raw reply
* Re: [RFC PATCH ghak32 V2 11/13] audit: add support for containerid to network namespaces
From: Richard Guy Briggs @ 2018-04-20 20:02 UTC (permalink / raw)
To: Paul Moore
Cc: cgroups, containers, linux-api, Linux-Audit Mailing List,
linux-fsdevel, LKML, netdev, ebiederm, luto, jlayton, carlos,
dhowells, viro, simo, Eric Paris, serge
In-Reply-To: <CAHC9VhRkstDMjd5T3w+iOUDjzDAs1AOm0xd3p6v_xn6fNGYQhA@mail.gmail.com>
On 2018-04-18 21:46, Paul Moore wrote:
> On Fri, Mar 16, 2018 at 5:00 AM, Richard Guy Briggs <rgb@redhat.com> wrote:
> > Audit events could happen in a network namespace outside of a task
> > context due to packets received from the net that trigger an auditing
> > rule prior to being associated with a running task. The network
> > namespace could in use by multiple containers by association to the
> > tasks in that network namespace. We still want a way to attribute
> > these events to any potential containers. Keep a list per network
> > namespace to track these container identifiiers.
> >
> > Add/increment the container identifier on:
> > - initial setting of the container id via /proc
> > - clone/fork call that inherits a container identifier
> > - unshare call that inherits a container identifier
> > - setns call that inherits a container identifier
> > Delete/decrement the container identifier on:
> > - an inherited container id dropped when child set
> > - process exit
> > - unshare call that drops a net namespace
> > - setns call that drops a net namespace
> >
> > See: https://github.com/linux-audit/audit-kernel/issues/32
> > See: https://github.com/linux-audit/audit-testsuite/issues/64
> > Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> > ---
> > include/linux/audit.h | 7 +++++++
> > include/net/net_namespace.h | 12 ++++++++++++
> > kernel/auditsc.c | 9 ++++++---
> > kernel/nsproxy.c | 6 ++++++
> > net/core/net_namespace.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
> > 5 files changed, 76 insertions(+), 3 deletions(-)
>
> ...
>
> > diff --git a/include/net/net_namespace.h b/include/net/net_namespace.h
> > index 0490084..343a428 100644
> > --- a/include/net/net_namespace.h
> > +++ b/include/net/net_namespace.h
> > @@ -33,6 +33,7 @@
> > #include <linux/ns_common.h>
> > #include <linux/idr.h>
> > #include <linux/skbuff.h>
> > +#include <linux/audit.h>
> >
> > struct user_namespace;
> > struct proc_dir_entry;
> > @@ -150,6 +151,7 @@ struct net {
> > #endif
> > struct sock *diag_nlsk;
> > atomic_t fnhe_genid;
> > + struct list_head audit_containerid;
> > } __randomize_layout;
>
> We talked about this briefly off-list, you should be using audit_net
> and the net_generic mechanism instead of this.
>
> > #include <linux/seq_file_net.h>
> > @@ -301,6 +303,16 @@ static inline struct net *read_pnet(const possible_net_t *pnet)
> > #define __net_initconst __initconst
> > #endif
> >
> > +#ifdef CONFIG_NET_NS
> > +void net_add_audit_containerid(struct net *net, u64 containerid);
> > +void net_del_audit_containerid(struct net *net, u64 containerid);
> > +#else
> > +static inline void net_add_audit_containerid(struct net *, u64)
> > +{ }
> > +static inline void net_del_audit_containerid(struct net *, u64)
> > +{ }
> > +#endif
> > +
> > int peernet2id_alloc(struct net *net, struct net *peer);
> > int peernet2id(struct net *net, struct net *peer);
> > bool peernet_has_id(struct net *net, struct net *peer);
> > diff --git a/kernel/auditsc.c b/kernel/auditsc.c
> > index 2f02ed9..208da962 100644
> > --- a/kernel/auditsc.c
> > +++ b/kernel/auditsc.c
> > @@ -75,6 +75,7 @@
> > #include <linux/uaccess.h>
> > #include <linux/fsnotify_backend.h>
> > #include <uapi/linux/limits.h>
> > +#include <net/net_namespace.h>
> >
> > #include "audit.h"
> >
> > @@ -2175,16 +2176,18 @@ static void audit_log_set_containerid(struct task_struct *task, u64 oldcontainer
> > */
> > int audit_set_containerid(struct task_struct *task, u64 containerid)
> > {
> > - u64 oldcontainerid;
> > + u64 oldcontainerid = audit_get_containerid(task);
> > int rc;
> > -
> > - oldcontainerid = audit_get_containerid(task);
> > + struct net *net = task->nsproxy->net_ns;
> >
> > rc = audit_set_containerid_perm(task, containerid);
> > if (!rc) {
> > + if (cid_valid(oldcontainerid))
> > + net_del_audit_containerid(net, oldcontainerid);
>
> Using audit_net we can handle this internal to audit, which is a Good Thing.
No problem, done.
> > task_lock(task);
> > task->containerid = containerid;
> > task_unlock(task);
> > + net_add_audit_containerid(net, containerid);
>
> Same.
>
> > }
> >
> > audit_log_set_containerid(task, oldcontainerid, containerid, rc);
> > diff --git a/kernel/nsproxy.c b/kernel/nsproxy.c
> > index f6c5d33..d9f1090 100644
> > --- a/kernel/nsproxy.c
> > +++ b/kernel/nsproxy.c
> > @@ -140,6 +140,7 @@ int copy_namespaces(unsigned long flags, struct task_struct *tsk)
> > struct nsproxy *old_ns = tsk->nsproxy;
> > struct user_namespace *user_ns = task_cred_xxx(tsk, user_ns);
> > struct nsproxy *new_ns;
> > + u64 containerid = audit_get_containerid(tsk);
> >
> > if (likely(!(flags & (CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWIPC |
> > CLONE_NEWPID | CLONE_NEWNET |
> > @@ -167,6 +168,7 @@ int copy_namespaces(unsigned long flags, struct task_struct *tsk)
> > return PTR_ERR(new_ns);
> >
> > tsk->nsproxy = new_ns;
> > + net_add_audit_containerid(new_ns->net_ns, containerid);
> > return 0;
> > }
>
> Hopefully we can handle this in audit_net_init(), we just need to
> figure out where we can get the correct task_struct for the audit
> container ID (some backpointer in the net struct?).
I don't follow. This needs to happen on every task startup.
audit_net_init() is only called when a new network namespace starts up.
> > @@ -217,6 +219,7 @@ int unshare_nsproxy_namespaces(unsigned long unshare_flags,
> > void switch_task_namespaces(struct task_struct *p, struct nsproxy *new)
> > {
> > struct nsproxy *ns;
> > + u64 containerid = audit_get_containerid(p);
> >
> > might_sleep();
> >
> > @@ -224,6 +227,9 @@ void switch_task_namespaces(struct task_struct *p, struct nsproxy *new)
> > ns = p->nsproxy;
> > p->nsproxy = new;
> > task_unlock(p);
> > + net_del_audit_containerid(ns->net_ns, containerid);
> > + if (new)
> > + net_add_audit_containerid(new->net_ns, containerid);
>
> Okay, we might need a hook here for switching namespaces, but I would
> much rather it be a generic audit hook that calls directly into audit.
Trivial, done.
> paul moore
- RGB
--
Richard Guy Briggs <rgb@redhat.com>
Sr. S/W Engineer, Kernel Security, Base Operating Systems
Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox