[PATCH net-next 0/2] allow setting gso_maximum values

[PATCH net-next 0/2] allow setting gso_maximum values

[Stephen Hemminger]

This is another way of addressing the GSO maximum performance issues for
containers on Azure. What happens is that the underlying infrastructure uses
a overlay network such that GSO packets over 64K - vlan header end up cause
either guest or host to have do expensive software copy and fragmentation.

The netvsc driver reports GSO maximum settings correctly, the issue
is that containers on veth devices still have the larger settings.
One solution that was examined was propogating the values back
through the bridge device, but this does not work for cases where
virtual container network is done on L3.

This patch set punts the problem to the orchestration layer that sets
up the container network. It also enables other virtual devices
to have configurable settings for GSO maximum.

Stephen Hemminger (2):
  rtnetlink: allow GSO maximums to be passed to device
  veth: allow configuring GSO maximums

 drivers/net/veth.c   | 20 ++++++++++++++++++++
 net/core/rtnetlink.c |  2 ++
 2 files changed, 22 insertions(+)

-- 
2.11.0

[PATCH net-next 1/2] rtnetlink: allow GSO maximums to be passed to device

[Stephen Hemminger]

Allow GSO maximum segments and size as netlink parameters on input.

Signed-off-by: Stephen Hemminger <sthemmin@microsoft.com>
---
 net/core/rtnetlink.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
index dabba2a91fc8..8138194c5f81 100644
--- a/net/core/rtnetlink.c
+++ b/net/core/rtnetlink.c
@@ -1569,6 +1569,8 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = {
 	[IFLA_PROMISCUITY]	= { .type = NLA_U32 },
 	[IFLA_NUM_TX_QUEUES]	= { .type = NLA_U32 },
 	[IFLA_NUM_RX_QUEUES]	= { .type = NLA_U32 },
+	[IFLA_GSO_MAX_SEGS]	= { .type = NLA_U32 },
+	[IFLA_GSO_MAX_SIZE]	= { .type = NLA_U32 },
 	[IFLA_PHYS_PORT_ID]	= { .type = NLA_BINARY, .len = MAX_PHYS_ITEM_ID_LEN },
 	[IFLA_CARRIER_CHANGES]	= { .type = NLA_U32 },  /* ignored */
 	[IFLA_PHYS_SWITCH_ID]	= { .type = NLA_BINARY, .len = MAX_PHYS_ITEM_ID_LEN },
-- 
2.11.0

[PATCH net-next 2/2] veth: allow configuring GSO maximums

[Stephen Hemminger]

Veth's can be used in environments (like Azure) where the underlying
network device is impacted by large GSO packets. This patch allows
gso maximum values to be passed in when creating the device via
netlink.

In theory, other pseudo devices could also use netlink attributes
to set GSO maximums but for now veth is what has been observed
to be an issue.

Signed-off-by: Stephen Hemminger <sthemmin@microsoft.com>
---
 drivers/net/veth.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index f5438d0978ca..510c058ba227 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -410,6 +410,26 @@ static int veth_newlink(struct net *src_net, struct net_device *dev,
 	if (ifmp && (dev->ifindex != 0))
 		peer->ifindex = ifmp->ifi_index;
 
+	if (tbp[IFLA_GSO_MAX_SIZE]) {
+		u32 max_size = nla_get_u32(tbp[IFLA_GSO_MAX_SIZE]);
+
+		if (max_size > GSO_MAX_SIZE)
+			return -EINVAL;
+
+		peer->gso_max_size = max_size;
+		dev->gso_max_size = max_size;
+	}
+
+	if (tbp[IFLA_GSO_MAX_SEGS]) {
+		u32 max_segs = nla_get_u32(tbp[IFLA_GSO_MAX_SEGS]);
+
+		if (max_segs > GSO_MAX_SEGS)
+			return -EINVAL;
+
+		peer->gso_max_segs = max_segs;
+		dev->gso_max_segs = max_segs;
+	}
+
 	err = register_netdevice(peer);
 	put_net(net);
 	net = NULL;
-- 
2.11.0

Re: [PATCH net-next 0/2] allow setting gso_maximum values

[Stephen Hemminger]

On Fri,  1 Dec 2017 12:11:56 -0800
Stephen Hemminger <stephen@networkplumber.org> wrote:

> This is another way of addressing the GSO maximum performance issues for
> containers on Azure. What happens is that the underlying infrastructure uses
> a overlay network such that GSO packets over 64K - vlan header end up cause
> either guest or host to have do expensive software copy and fragmentation.
> 
> The netvsc driver reports GSO maximum settings correctly, the issue
> is that containers on veth devices still have the larger settings.
> One solution that was examined was propogating the values back
> through the bridge device, but this does not work for cases where
> virtual container network is done on L3.
> 
> This patch set punts the problem to the orchestration layer that sets
> up the container network. It also enables other virtual devices
> to have configurable settings for GSO maximum.
> 
> Stephen Hemminger (2):
>   rtnetlink: allow GSO maximums to be passed to device
>   veth: allow configuring GSO maximums
> 
>  drivers/net/veth.c   | 20 ++++++++++++++++++++
>  net/core/rtnetlink.c |  2 ++
>  2 files changed, 22 insertions(+)
> 

I would like a confirmation from Intel that is doing Docker testing
that this works for them before merging.

Re: [PATCH net-next 0/2] allow setting gso_maximum values

[Solio Sarabia]

On Fri, Dec 01, 2017 at 03:30:01PM -0800, Stephen Hemminger wrote:
> On Fri,  1 Dec 2017 12:11:56 -0800
> Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> > This is another way of addressing the GSO maximum performance issues for
> > containers on Azure. What happens is that the underlying infrastructure uses
> > a overlay network such that GSO packets over 64K - vlan header end up cause
> > either guest or host to have do expensive software copy and fragmentation.
> > 
> > The netvsc driver reports GSO maximum settings correctly, the issue
> > is that containers on veth devices still have the larger settings.
> > One solution that was examined was propogating the values back
> > through the bridge device, but this does not work for cases where
> > virtual container network is done on L3.
> > 
> > This patch set punts the problem to the orchestration layer that sets
> > up the container network. It also enables other virtual devices
> > to have configurable settings for GSO maximum.
> > 
> > Stephen Hemminger (2):
> >   rtnetlink: allow GSO maximums to be passed to device
> >   veth: allow configuring GSO maximums
> > 
> >  drivers/net/veth.c   | 20 ++++++++++++++++++++
> >  net/core/rtnetlink.c |  2 ++
> >  2 files changed, 22 insertions(+)
> > 
> 
> I would like a confirmation from Intel that is doing Docker testing
> that this works for them before merging.

This change and its iproute2 counterpart allow creating veth pairs with
specific gso_max{size,segs}. Thanks.

However, the docker code that sets up veth pairis is go-compiled in
their libnetwork. End-users won't be able to tweak gso settings at veth
creation. In this case, we would need to add ioctl (ip/iplink.c:do_set)
support to allow changes after veth is created.

Re: [PATCH net-next 2/2] veth: allow configuring GSO maximums

[David Ahern]

On 12/1/17 1:11 PM, Stephen Hemminger wrote:
> diff --git a/drivers/net/veth.c b/drivers/net/veth.c
> index f5438d0978ca..510c058ba227 100644
> --- a/drivers/net/veth.c
> +++ b/drivers/net/veth.c
> @@ -410,6 +410,26 @@ static int veth_newlink(struct net *src_net, struct net_device *dev,
>  	if (ifmp && (dev->ifindex != 0))
>  		peer->ifindex = ifmp->ifi_index;
>  
> +	if (tbp[IFLA_GSO_MAX_SIZE]) {
> +		u32 max_size = nla_get_u32(tbp[IFLA_GSO_MAX_SIZE]);
> +
> +		if (max_size > GSO_MAX_SIZE)
> +			return -EINVAL;
> +
> +		peer->gso_max_size = max_size;
> +		dev->gso_max_size = max_size;
> +	}
> +
> +	if (tbp[IFLA_GSO_MAX_SEGS]) {
> +		u32 max_segs = nla_get_u32(tbp[IFLA_GSO_MAX_SEGS]);
> +
> +		if (max_segs > GSO_MAX_SEGS)
> +			return -EINVAL;
> +
> +		peer->gso_max_segs = max_segs;
> +		dev->gso_max_segs = max_segs;
> +	}
> +
>  	err = register_netdevice(peer);
>  	put_net(net);
>  	net = NULL;
> 

Given the role of veth it seems odd to only allow these to be set at
create time. I think Solio is saying the same with respect to the Docker
use case.

Re: [PATCH net-next 0/2] allow setting gso_maximum values

[David Miller]

From: Stephen Hemminger <stephen@networkplumber.org>
Date: Fri, 1 Dec 2017 15:30:01 -0800

> On Fri,  1 Dec 2017 12:11:56 -0800
> Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
>> This is another way of addressing the GSO maximum performance issues for
>> containers on Azure. What happens is that the underlying infrastructure uses
>> a overlay network such that GSO packets over 64K - vlan header end up cause
>> either guest or host to have do expensive software copy and fragmentation.
>> 
>> The netvsc driver reports GSO maximum settings correctly, the issue
>> is that containers on veth devices still have the larger settings.
>> One solution that was examined was propogating the values back
>> through the bridge device, but this does not work for cases where
>> virtual container network is done on L3.
>> 
>> This patch set punts the problem to the orchestration layer that sets
>> up the container network. It also enables other virtual devices
>> to have configurable settings for GSO maximum.
>> 
>> Stephen Hemminger (2):
>>   rtnetlink: allow GSO maximums to be passed to device
>>   veth: allow configuring GSO maximums
>> 
>>  drivers/net/veth.c   | 20 ++++++++++++++++++++
>>  net/core/rtnetlink.c |  2 ++
>>  2 files changed, 22 insertions(+)
>> 
> 
> I would like a confirmation from Intel that is doing Docker testing
> that this works for them before merging.

Like David Ahern, I think you should allow this net netlink setting
during changelink as well as newlink.

Thanks.

Re: [PATCH net-next 2/2] veth: allow configuring GSO maximums

[Cong Wang]

On Fri, Dec 1, 2017 at 12:11 PM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> Veth's can be used in environments (like Azure) where the underlying
> network device is impacted by large GSO packets. This patch allows
> gso maximum values to be passed in when creating the device via
> netlink.
>
> In theory, other pseudo devices could also use netlink attributes
> to set GSO maximums but for now veth is what has been observed
> to be an issue.

It looks odd that you only allow setting them but not dumping them.

Re: [PATCH net-next 2/2] veth: allow configuring GSO maximums

[Eric Dumazet]

On Mon, 2017-12-04 at 16:03 -0800, Cong Wang wrote:
> On Fri, Dec 1, 2017 at 12:11 PM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> > Veth's can be used in environments (like Azure) where the
> > underlying
> > network device is impacted by large GSO packets. This patch allows
> > gso maximum values to be passed in when creating the device via
> > netlink.
> > 
> > In theory, other pseudo devices could also use netlink attributes
> > to set GSO maximums but for now veth is what has been observed
> > to be an issue.
> 
> It looks odd that you only allow setting them but not dumping them.

Dump is already supported/provided by core stack

# ip -d link sh dev wlan0
2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP
mode DORMANT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff promiscuity 0
addrgenmode none numtxqueues 4 numrxqueues 1 gso_max_size 65536
gso_max_segs 65535