[PATCH net-next v2 0/6] net: Add STT support.

[PATCH net-next v2 0/6] net: Add STT support.

[Pravin B Shelar]

Following patch series adds support for Stateless Transport
Tunneling protocol.
STT uses TCP segmentation offload available in most of NIC. On
packet xmit STT driver appends STT header along with TCP header
to the packet. For GSO packet GSO parameters are set according
to tunnel configuration and packet is handed over to networking
stack. This allows use of segmentation offload available in NICs

The protocol is documented at
http://www.ietf.org/archive/id/draft-davie-stt-06.txt

I will send out OVS userspace patch on ovs-dev mailing list.

Following are test results. All tests are done on net-next with
STT and VXLAN kernel device without OVS.

Single Netperf session:
=======================
VXLAN:
    CPU utilization
     - Send local: 1.26
     - Recv remote: 8.62
    Throughput: 4.9 Gbit/sec
STT:
    CPU utilization
     - Send local: 1.01
     - Recv remote: 1.8
    Throughput: 9.45 Gbit/sec

Five Netperf sessions:
======================
VXLAN:
    CPU utilization
     - Send local: 9.7
     - Recv remote: 70 (varies from 60 to 80)
    Throughput: 9.05 Gbit/sec
STT:
    CPU utilization
     - Send local: 5.85
     - Recv remote: 14
    Throughput: 9.47 Gbit/sec

v1-v2:
- Added STT device.
- Fixed first patch according to comments from Eric.

Pravin B Shelar (6):
  skbuff: Add skb_list_linearize()
  net: Add STT tunneling protocol.
  openvswitch: Add support for STT tunneling.
  ip_tunnel: Introduce tunnel_info struct.
  ip_tunnel: Extend tunnel_info for STT.
  net: Add STT device.

 include/linux/skbuff.h           |    2 +
 include/net/ip_tunnels.h         |   63 +-
 include/net/stt.h                |   59 ++
 include/uapi/linux/if_tunnel.h   |   17 +
 include/uapi/linux/openvswitch.h |    1 +
 net/core/skbuff.c                |   34 +
 net/ipv4/Kconfig                 |   20 +
 net/ipv4/Makefile                |    2 +
 net/ipv4/gre_demux.c             |    4 +-
 net/ipv4/ip_gre.c                |  183 +++--
 net/ipv4/ip_stt.c                |  410 +++++++++++
 net/ipv4/ip_tunnel.c             |  312 +++++----
 net/ipv4/ip_vti.c                |   75 +-
 net/ipv4/ipip.c                  |   88 +--
 net/ipv4/stt.c                   | 1398 ++++++++++++++++++++++++++++++++++++++
 net/ipv6/sit.c                   |  152 ++---
 net/openvswitch/Kconfig          |   10 +
 net/openvswitch/Makefile         |    1 +
 net/openvswitch/vport-stt.c      |  216 ++++++
 19 files changed, 2674 insertions(+), 373 deletions(-)
 create mode 100644 include/net/stt.h
 create mode 100644 net/ipv4/ip_stt.c
 create mode 100644 net/ipv4/stt.c
 create mode 100644 net/openvswitch/vport-stt.c

-- 
1.9.1

Re: [PATCH net-next v2 0/6] net: Add STT support.

[Alexander Duyck]

On 01/29/2015 03:29 PM, Pravin B Shelar wrote:
> Following patch series adds support for Stateless Transport
> Tunneling protocol.
> STT uses TCP segmentation offload available in most of NIC. On
> packet xmit STT driver appends STT header along with TCP header
> to the packet. For GSO packet GSO parameters are set according
> to tunnel configuration and packet is handed over to networking
> stack. This allows use of segmentation offload available in NICs
>
> The protocol is documented at
> http://www.ietf.org/archive/id/draft-davie-stt-06.txt
>
> I will send out OVS userspace patch on ovs-dev mailing list.
>
> Following are test results. All tests are done on net-next with
> STT and VXLAN kernel device without OVS.
>
> Single Netperf session:
> =======================
> VXLAN:
>     CPU utilization
>      - Send local: 1.26
>      - Recv remote: 8.62
>     Throughput: 4.9 Gbit/sec
> STT:
>     CPU utilization
>      - Send local: 1.01
>      - Recv remote: 1.8
>     Throughput: 9.45 Gbit/sec
>
> Five Netperf sessions:
> ======================
> VXLAN:
>     CPU utilization
>      - Send local: 9.7
>      - Recv remote: 70 (varies from 60 to 80)
>     Throughput: 9.05 Gbit/sec
> STT:
>     CPU utilization
>      - Send local: 5.85
>      - Recv remote: 14
>     Throughput: 9.47 Gbit/sec
>

What does the small packet or non-TCP performance look like for STT vs
VXLAN?  My concern is that STT looks like it is a one trick pony since
all your numbers show is TCP TSO performance, and based on some of the
comments in your patches it seems like other protocols such as UDP are
going to suffer pretty badly due to things like the linearization overhead.

- Alex

Re: [PATCH net-next v2 0/6] net: Add STT support.

[Pravin Shelar]

On Thu, Jan 29, 2015 at 7:46 PM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:
> On 01/29/2015 03:29 PM, Pravin B Shelar wrote:
>> Following patch series adds support for Stateless Transport
>> Tunneling protocol.
>> STT uses TCP segmentation offload available in most of NIC. On
>> packet xmit STT driver appends STT header along with TCP header
>> to the packet. For GSO packet GSO parameters are set according
>> to tunnel configuration and packet is handed over to networking
>> stack. This allows use of segmentation offload available in NICs
>>
>> The protocol is documented at
>> http://www.ietf.org/archive/id/draft-davie-stt-06.txt
>>
>> I will send out OVS userspace patch on ovs-dev mailing list.
>>
>> Following are test results. All tests are done on net-next with
>> STT and VXLAN kernel device without OVS.
>>
>> Single Netperf session:
>> =======================
>> VXLAN:
>>     CPU utilization
>>      - Send local: 1.26
>>      - Recv remote: 8.62
>>     Throughput: 4.9 Gbit/sec
>> STT:
>>     CPU utilization
>>      - Send local: 1.01
>>      - Recv remote: 1.8
>>     Throughput: 9.45 Gbit/sec
>>
>> Five Netperf sessions:
>> ======================
>> VXLAN:
>>     CPU utilization
>>      - Send local: 9.7
>>      - Recv remote: 70 (varies from 60 to 80)
>>     Throughput: 9.05 Gbit/sec
>> STT:
>>     CPU utilization
>>      - Send local: 5.85
>>      - Recv remote: 14
>>     Throughput: 9.47 Gbit/sec
>>
>
> What does the small packet or non-TCP performance look like for STT vs
> VXLAN?  My concern is that STT looks like it is a one trick pony since
> all your numbers show is TCP TSO performance, and based on some of the
> comments in your patches it seems like other protocols such as UDP are
> going to suffer pretty badly due to things like the linearization overhead.
>

Current implementation is targeted for TCP workloads thats why I
posted numbers with TCP, once UDP is optimized we can discuss UDP
numbers. I am pretty sure the STT code can be optimized further
specially for protocols other than TCP.

Re: [PATCH net-next v2 0/6] net: Add STT support.

[Tom Herbert]

On Thu, Jan 29, 2015 at 8:04 PM, Pravin Shelar <pshelar@nicira.com> wrote:
> On Thu, Jan 29, 2015 at 7:46 PM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>> On 01/29/2015 03:29 PM, Pravin B Shelar wrote:
>>> Following patch series adds support for Stateless Transport
>>> Tunneling protocol.
>>> STT uses TCP segmentation offload available in most of NIC. On
>>> packet xmit STT driver appends STT header along with TCP header
>>> to the packet. For GSO packet GSO parameters are set according
>>> to tunnel configuration and packet is handed over to networking
>>> stack. This allows use of segmentation offload available in NICs
>>>
>>> The protocol is documented at
>>> http://www.ietf.org/archive/id/draft-davie-stt-06.txt
>>>
>>> I will send out OVS userspace patch on ovs-dev mailing list.
>>>
>>> Following are test results. All tests are done on net-next with
>>> STT and VXLAN kernel device without OVS.
>>>
>>> Single Netperf session:
>>> =======================
>>> VXLAN:
>>>     CPU utilization
>>>      - Send local: 1.26
>>>      - Recv remote: 8.62
>>>     Throughput: 4.9 Gbit/sec
>>> STT:
>>>     CPU utilization
>>>      - Send local: 1.01
>>>      - Recv remote: 1.8
>>>     Throughput: 9.45 Gbit/sec
>>>
>>> Five Netperf sessions:
>>> ======================
>>> VXLAN:
>>>     CPU utilization
>>>      - Send local: 9.7
>>>      - Recv remote: 70 (varies from 60 to 80)
>>>     Throughput: 9.05 Gbit/sec
>>> STT:
>>>     CPU utilization
>>>      - Send local: 5.85
>>>      - Recv remote: 14
>>>     Throughput: 9.47 Gbit/sec
>>>
>>
>> What does the small packet or non-TCP performance look like for STT vs
>> VXLAN?  My concern is that STT looks like it is a one trick pony since
>> all your numbers show is TCP TSO performance, and based on some of the
>> comments in your patches it seems like other protocols such as UDP are
>> going to suffer pretty badly due to things like the linearization overhead.
>>
>
> Current implementation is targeted for TCP workloads thats why I
> posted numbers with TCP, once UDP is optimized we can discuss UDP
> numbers. I am pretty sure the STT code can be optimized further
> specially for protocols other than TCP.
> --
There are many TCP workloads that use small packets, it is critical to
test for these also. E.g. "super_netperf 200 -H <addr> -l 120 -t
TCP_RR -- -r 1,1"

Please provide the *exact* commands that you are using to configure
stt for optimal performance.

Tom

> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net-next v2 0/6] net: Add STT support.

[Pravin Shelar]

On Thu, Jan 29, 2015 at 8:17 PM, Tom Herbert <therbert@google.com> wrote:
> On Thu, Jan 29, 2015 at 8:04 PM, Pravin Shelar <pshelar@nicira.com> wrote:
>> On Thu, Jan 29, 2015 at 7:46 PM, Alexander Duyck
>> <alexander.duyck@gmail.com> wrote:
>>> On 01/29/2015 03:29 PM, Pravin B Shelar wrote:
>>>> Following patch series adds support for Stateless Transport
>>>> Tunneling protocol.
>>>> STT uses TCP segmentation offload available in most of NIC. On
>>>> packet xmit STT driver appends STT header along with TCP header
>>>> to the packet. For GSO packet GSO parameters are set according
>>>> to tunnel configuration and packet is handed over to networking
>>>> stack. This allows use of segmentation offload available in NICs
>>>>
>>>> The protocol is documented at
>>>> http://www.ietf.org/archive/id/draft-davie-stt-06.txt
>>>>
>>>> I will send out OVS userspace patch on ovs-dev mailing list.
>>>>
>>>> Following are test results. All tests are done on net-next with
>>>> STT and VXLAN kernel device without OVS.
>>>>
>>>> Single Netperf session:
>>>> =======================
>>>> VXLAN:
>>>>     CPU utilization
>>>>      - Send local: 1.26
>>>>      - Recv remote: 8.62
>>>>     Throughput: 4.9 Gbit/sec
>>>> STT:
>>>>     CPU utilization
>>>>      - Send local: 1.01
>>>>      - Recv remote: 1.8
>>>>     Throughput: 9.45 Gbit/sec
>>>>
>>>> Five Netperf sessions:
>>>> ======================
>>>> VXLAN:
>>>>     CPU utilization
>>>>      - Send local: 9.7
>>>>      - Recv remote: 70 (varies from 60 to 80)
>>>>     Throughput: 9.05 Gbit/sec
>>>> STT:
>>>>     CPU utilization
>>>>      - Send local: 5.85
>>>>      - Recv remote: 14
>>>>     Throughput: 9.47 Gbit/sec
>>>>
>>>
>>> What does the small packet or non-TCP performance look like for STT vs
>>> VXLAN?  My concern is that STT looks like it is a one trick pony since
>>> all your numbers show is TCP TSO performance, and based on some of the
>>> comments in your patches it seems like other protocols such as UDP are
>>> going to suffer pretty badly due to things like the linearization overhead.
>>>
>>
>> Current implementation is targeted for TCP workloads thats why I
>> posted numbers with TCP, once UDP is optimized we can discuss UDP
>> numbers. I am pretty sure the STT code can be optimized further
>> specially for protocols other than TCP.
>> --
> There are many TCP workloads that use small packets, it is critical to
> test for these also. E.g. "super_netperf 200 -H <addr> -l 120 -t
> TCP_RR -- -r 1,1"
>
I have not tried it on STT device, I will collect those numbers.

> Please provide the *exact* commands that you are using to configure
> stt for optimal performance.
>
To create STT tunnel device.
`ip link add stt1  type stt key 1 remote 1.1.2.128`

No other configuration is needed.

Re: [PATCH net-next v2 0/6] net: Add STT support.

[Andy Gospodarek]

On Thu, Jan 29, 2015 at 09:03:14PM -0800, Pravin Shelar wrote:
> On Thu, Jan 29, 2015 at 8:17 PM, Tom Herbert <therbert@google.com> wrote:
> > On Thu, Jan 29, 2015 at 8:04 PM, Pravin Shelar <pshelar@nicira.com> wrote:
> >> On Thu, Jan 29, 2015 at 7:46 PM, Alexander Duyck
> >> <alexander.duyck@gmail.com> wrote:
> >>> On 01/29/2015 03:29 PM, Pravin B Shelar wrote:
> >>>> Following patch series adds support for Stateless Transport
> >>>> Tunneling protocol.
> >>>> STT uses TCP segmentation offload available in most of NIC. On
> >>>> packet xmit STT driver appends STT header along with TCP header
> >>>> to the packet. For GSO packet GSO parameters are set according
> >>>> to tunnel configuration and packet is handed over to networking
> >>>> stack. This allows use of segmentation offload available in NICs
> >>>>
> >>>> The protocol is documented at
> >>>> http://www.ietf.org/archive/id/draft-davie-stt-06.txt
> >>>>
> >>>> I will send out OVS userspace patch on ovs-dev mailing list.
> >>>>
> >>>> Following are test results. All tests are done on net-next with
> >>>> STT and VXLAN kernel device without OVS.
> >>>>
> >>>> Single Netperf session:
> >>>> =======================
> >>>> VXLAN:
> >>>>     CPU utilization
> >>>>      - Send local: 1.26
> >>>>      - Recv remote: 8.62
> >>>>     Throughput: 4.9 Gbit/sec
> >>>> STT:
> >>>>     CPU utilization
> >>>>      - Send local: 1.01
> >>>>      - Recv remote: 1.8
> >>>>     Throughput: 9.45 Gbit/sec
> >>>>
> >>>> Five Netperf sessions:
> >>>> ======================
> >>>> VXLAN:
> >>>>     CPU utilization
> >>>>      - Send local: 9.7
> >>>>      - Recv remote: 70 (varies from 60 to 80)
> >>>>     Throughput: 9.05 Gbit/sec
> >>>> STT:
> >>>>     CPU utilization
> >>>>      - Send local: 5.85
> >>>>      - Recv remote: 14
> >>>>     Throughput: 9.47 Gbit/sec
> >>>>
> >>>
> >>> What does the small packet or non-TCP performance look like for STT vs
> >>> VXLAN?  My concern is that STT looks like it is a one trick pony since
> >>> all your numbers show is TCP TSO performance, and based on some of the
> >>> comments in your patches it seems like other protocols such as UDP are
> >>> going to suffer pretty badly due to things like the linearization overhead.
> >>>
> >>
> >> Current implementation is targeted for TCP workloads thats why I
> >> posted numbers with TCP, once UDP is optimized we can discuss UDP
> >> numbers. I am pretty sure the STT code can be optimized further
> >> specially for protocols other than TCP.
> >> --
> > There are many TCP workloads that use small packets, it is critical to
> > test for these also. E.g. "super_netperf 200 -H <addr> -l 120 -t
> > TCP_RR -- -r 1,1"
> >
> I have not tried it on STT device, I will collect those numbers.
> 
> > Please provide the *exact* commands that you are using to configure
> > stt for optimal performance.
> >
> To create STT tunnel device.
> `ip link add stt1  type stt key 1 remote 1.1.2.128`
> 
> No other configuration is needed.

Thanks for posting some performance numbers with your patch.  I also
don't want to 'pile on' with additional complaints, but I do have one
request.

Can you share any specs (including number of cores and NIC hardware
used) for the systems that gave you the above results?   If you do not
want to endorse a particular NIC that is fine --  I'm mostly curious how
many cores were used and if UDP and TCP RSS were both being used in this
configuration.

Thanks!

Re: [PATCH net-next v2 0/6] net: Add STT support.

[Pravin Shelar]

On Fri, Jan 30, 2015 at 10:44 AM, Andy Gospodarek
<gospo@cumulusnetworks.com> wrote:
> On Thu, Jan 29, 2015 at 09:03:14PM -0800, Pravin Shelar wrote:
>> On Thu, Jan 29, 2015 at 8:17 PM, Tom Herbert <therbert@google.com> wrote:
>> > On Thu, Jan 29, 2015 at 8:04 PM, Pravin Shelar <pshelar@nicira.com> wrote:
>> >> On Thu, Jan 29, 2015 at 7:46 PM, Alexander Duyck
>> >> <alexander.duyck@gmail.com> wrote:
>> >>> On 01/29/2015 03:29 PM, Pravin B Shelar wrote:
>> >>>> Following patch series adds support for Stateless Transport
>> >>>> Tunneling protocol.
>> >>>> STT uses TCP segmentation offload available in most of NIC. On
>> >>>> packet xmit STT driver appends STT header along with TCP header
>> >>>> to the packet. For GSO packet GSO parameters are set according
>> >>>> to tunnel configuration and packet is handed over to networking
>> >>>> stack. This allows use of segmentation offload available in NICs
>> >>>>
>> >>>> The protocol is documented at
>> >>>> http://www.ietf.org/archive/id/draft-davie-stt-06.txt
>> >>>>
>> >>>> I will send out OVS userspace patch on ovs-dev mailing list.
>> >>>>
>> >>>> Following are test results. All tests are done on net-next with
>> >>>> STT and VXLAN kernel device without OVS.
>> >>>>
>> >>>> Single Netperf session:
>> >>>> =======================
>> >>>> VXLAN:
>> >>>>     CPU utilization
>> >>>>      - Send local: 1.26
>> >>>>      - Recv remote: 8.62
>> >>>>     Throughput: 4.9 Gbit/sec
>> >>>> STT:
>> >>>>     CPU utilization
>> >>>>      - Send local: 1.01
>> >>>>      - Recv remote: 1.8
>> >>>>     Throughput: 9.45 Gbit/sec
>> >>>>
>> >>>> Five Netperf sessions:
>> >>>> ======================
>> >>>> VXLAN:
>> >>>>     CPU utilization
>> >>>>      - Send local: 9.7
>> >>>>      - Recv remote: 70 (varies from 60 to 80)
>> >>>>     Throughput: 9.05 Gbit/sec
>> >>>> STT:
>> >>>>     CPU utilization
>> >>>>      - Send local: 5.85
>> >>>>      - Recv remote: 14
>> >>>>     Throughput: 9.47 Gbit/sec
>> >>>>
>> >>>
>> >>> What does the small packet or non-TCP performance look like for STT vs
>> >>> VXLAN?  My concern is that STT looks like it is a one trick pony since
>> >>> all your numbers show is TCP TSO performance, and based on some of the
>> >>> comments in your patches it seems like other protocols such as UDP are
>> >>> going to suffer pretty badly due to things like the linearization overhead.
>> >>>
>> >>
>> >> Current implementation is targeted for TCP workloads thats why I
>> >> posted numbers with TCP, once UDP is optimized we can discuss UDP
>> >> numbers. I am pretty sure the STT code can be optimized further
>> >> specially for protocols other than TCP.
>> >> --
>> > There are many TCP workloads that use small packets, it is critical to
>> > test for these also. E.g. "super_netperf 200 -H <addr> -l 120 -t
>> > TCP_RR -- -r 1,1"
>> >
>> I have not tried it on STT device, I will collect those numbers.
>>
>> > Please provide the *exact* commands that you are using to configure
>> > stt for optimal performance.
>> >
>> To create STT tunnel device.
>> `ip link add stt1  type stt key 1 remote 1.1.2.128`
>>
>> No other configuration is needed.
>
> Thanks for posting some performance numbers with your patch.  I also
> don't want to 'pile on' with additional complaints, but I do have one
> request.
>
> Can you share any specs (including number of cores and NIC hardware
> used) for the systems that gave you the above results?   If you do not
> want to endorse a particular NIC that is fine --  I'm mostly curious how
> many cores were used and if UDP and TCP RSS were both being used in this
> configuration.
>

I used 16 core (Sandy bridge) machine with intel 10G NIC. I enabled
both UDP and TCP RSS for all tests.

Re: [PATCH net-next v2 0/6] net: Add STT support.

[Rick Jones]

On 01/29/2015 08:04 PM, Pravin Shelar wrote:
> On Thu, Jan 29, 2015 at 7:46 PM, Alexander Duyck
>> What does the small packet or non-TCP performance look like for STT vs
>> VXLAN?  My concern is that STT looks like it is a one trick pony since
>> all your numbers show is TCP TSO performance, and based on some of the
>> comments in your patches it seems like other protocols such as UDP are
>> going to suffer pretty badly due to things like the linearization overhead.
>>
>
> Current implementation is targeted for TCP workloads thats why I
> posted numbers with TCP, once UDP is optimized we can discuss UDP
> numbers. I am pretty sure the STT code can be optimized further
> specially for protocols other than TCP.

Not to pile-on or anything but indeed, there is much more to "TCP 
workloads" than just bulk transfer (TCP_STREAM), which is precisely why 
netperf was created oh so many years ago with its TCP_RR test to try to 
replace ttcp, and why there are the methods of mine and others to do 
aggregate PPS with it and other benchmarks.

Of course that comment applies not only to STT but also to any other 
"get to link-rate" on a (single|smallnumberof) stream" change.

rick

Re: [PATCH net-next v2 0/6] net: Add STT support.

[David Miller]

From: Pravin B Shelar <pshelar@nicira.com>
Date: Thu, 29 Jan 2015 15:29:16 -0800

> Following patch series adds support for Stateless Transport
> Tunneling protocol.
> STT uses TCP segmentation offload available in most of NIC. On
> packet xmit STT driver appends STT header along with TCP header
> to the packet. For GSO packet GSO parameters are set according
> to tunnel configuration and packet is handed over to networking
> stack. This allows use of segmentation offload available in NICs

I don't like this at all.

Routers _absolutely_ depend upon the ability to make TCP flows back
off by dropping packets in various ways (tail drop, RED random drops,
etc.).  STT violates this completely.

It's _NOT_ TCP, you can't put lipstick on a pig and pretend it's not a
pig.  You need to use something that indicates it's flow properties,
a datagram protocol.  Either via an existing one or by creating a new
one.

I know you want to do this mass TCP behavioral violation because of
TSO.  But that's too bad.  The ends do not justify the means.

I also don't buy the argument that "people can put arbitrary changes
into their kernel to do stuff like that".

They can't do it to the stack I'm willing to maintain, and that's what
matters for %99 of systems out there.

Re: [PATCH net-next v2 0/6] net: Add STT support.

[Pravin Shelar]

On Sat, Jan 31, 2015 at 5:40 PM, David Miller <davem@davemloft.net> wrote:
> From: Pravin B Shelar <pshelar@nicira.com>
> Date: Thu, 29 Jan 2015 15:29:16 -0800
>
>> Following patch series adds support for Stateless Transport
>> Tunneling protocol.
>> STT uses TCP segmentation offload available in most of NIC. On
>> packet xmit STT driver appends STT header along with TCP header
>> to the packet. For GSO packet GSO parameters are set according
>> to tunnel configuration and packet is handed over to networking
>> stack. This allows use of segmentation offload available in NICs
>
> I don't like this at all.
>
> Routers _absolutely_ depend upon the ability to make TCP flows back
> off by dropping packets in various ways (tail drop, RED random drops,
> etc.).  STT violates this completely.
>
> It's _NOT_ TCP, you can't put lipstick on a pig and pretend it's not a
> pig.  You need to use something that indicates it's flow properties,
> a datagram protocol.  Either via an existing one or by creating a new
> one.
>
> I know you want to do this mass TCP behavioral violation because of
> TSO.  But that's too bad.  The ends do not justify the means.
>
> I also don't buy the argument that "people can put arbitrary changes
> into their kernel to do stuff like that".
>
> They can't do it to the stack I'm willing to maintain, and that's what
> matters for %99 of systems out there.

ok, I will drop patch series.