Poorer networking performance in later kernels?

netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* Poorer networking performance in later kernels?
       [not found] <SN1PR0301MB19987565B71A4688593A2CDBD6680@SN1PR0301MB1998.namprd03.prod.outlook.com>
@ 2016-04-15 21:02 ` Butler, Peter
  2016-04-15 22:33   ` Eric Dumazet
  2016-04-15 22:37   ` Rick Jones
  0 siblings, 2 replies; 16+ messages in thread
From: Butler, Peter @ 2016-04-15 21:02 UTC (permalink / raw)
  To: netdev@vger.kernel.org

(Please keep me CC'd to all comments/responses)

I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop in networking performance.  Nothing was changed on the test systems, other than the kernel itself (and kernel modules).  The identical .config used to build the 3.4.2 kernel was brought over into the 4.4.0 kernel source tree, and any configuration differences (e.g. new parameters, etc.) were taken as default values.

The testing was performed on the same actual hardware for both kernel versions (i.e. take the existing 3.4.2 physical setup, simply boot into the (new) kernel and run the same test).  The netperf utility was used for benchmarking and the testing was always performed on idle systems.

TCP testing yielded the following results, where the 4.4.0 kernel only got about 1/2 of the throughput:

      Recv     Send       Send                          Utilization       Service Demand
      Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
      Size     Size       Size    Time       Throughput local    remote   local   remote
      bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB

3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765

SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:

      Recv     Send       Send                          Utilization       Service Demand
      Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
      Size     Size       Size    Time       Throughput local    remote   local   remote
      bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB

3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210

The same tests were performed a multitude of time, and are always consistent (within a few percent).  I've also tried playing with various run-time kernel parameters (/proc/sys/kernel/net/...) on the 4.4.0 kernel to alleviate the issue but have had no success at all.

I'm at a loss as to what could possibly account for such a discrepancy...

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Poorer networking performance in later kernels?
  2016-04-15 21:02 ` Poorer networking performance in later kernels? Butler, Peter
@ 2016-04-15 22:33   ` Eric Dumazet
  2016-04-18 12:16     ` Eric Dumazet
  2016-04-15 22:37   ` Rick Jones
  1 sibling, 1 reply; 16+ messages in thread
From: Eric Dumazet @ 2016-04-15 22:33 UTC (permalink / raw)
  To: Butler, Peter; +Cc: netdev@vger.kernel.org

On Fri, 2016-04-15 at 21:02 +0000, Butler, Peter wrote:
> (Please keep me CC'd to all comments/responses)
> 
> I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop in networking performance.  Nothing was changed on the test systems, other than the kernel itself (and kernel modules).  The identical .config used to build the 3.4.2 kernel was brought over into the 4.4.0 kernel source tree, and any configuration differences (e.g. new parameters, etc.) were taken as default values.
> 
> The testing was performed on the same actual hardware for both kernel versions (i.e. take the existing 3.4.2 physical setup, simply boot into the (new) kernel and run the same test).  The netperf utility was used for benchmarking and the testing was always performed on idle systems.
> 
> TCP testing yielded the following results, where the 4.4.0 kernel only got about 1/2 of the throughput:
> 
>       Recv     Send       Send                          Utilization       Service Demand
>       Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>       Size     Size       Size    Time       Throughput local    remote   local   remote
>       bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
> 
> 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
> 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
> 
> SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
> 
>       Recv     Send       Send                          Utilization       Service Demand
>       Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>       Size     Size       Size    Time       Throughput local    remote   local   remote
>       bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
> 
> 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
> 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
> 
> The same tests were performed a multitude of time, and are always consistent (within a few percent).  I've also tried playing with various run-time kernel parameters (/proc/sys/kernel/net/...) on the 4.4.0 kernel to alleviate the issue but have had no success at all.
> 
> I'm at a loss as to what could possibly account for such a discrepancy...

Maybe new kernel is faster and you have drops somewhere ?

nstat >/dev/null
netperf -H ...
nstat

Would help

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Poorer networking performance in later kernels?
  2016-04-15 22:33   ` Eric Dumazet
@ 2016-04-18 12:16     ` Eric Dumazet
  2016-04-18 16:27       ` Butler, Peter
  2016-04-18 16:44       ` Butler, Peter
  0 siblings, 2 replies; 16+ messages in thread
From: Eric Dumazet @ 2016-04-18 12:16 UTC (permalink / raw)
  To: Butler, Peter; +Cc: netdev@vger.kernel.org

On Fri, 2016-04-15 at 15:33 -0700, Eric Dumazet wrote:
> On Fri, 2016-04-15 at 21:02 +0000, Butler, Peter wrote:
> > (Please keep me CC'd to all comments/responses)
> > 
> > I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop in networking performance.  Nothing was changed on the test systems, other than the kernel itself (and kernel modules).  The identical .config used to build the 3.4.2 kernel was brought over into the 4.4.0 kernel source tree, and any configuration differences (e.g. new parameters, etc.) were taken as default values.
> > 
> > The testing was performed on the same actual hardware for both kernel versions (i.e. take the existing 3.4.2 physical setup, simply boot into the (new) kernel and run the same test).  The netperf utility was used for benchmarking and the testing was always performed on idle systems.
> > 
> > TCP testing yielded the following results, where the 4.4.0 kernel only got about 1/2 of the throughput:
> > 
> >       Recv     Send       Send                          Utilization       Service Demand
> >       Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
> >       Size     Size       Size    Time       Throughput local    remote   local   remote
> >       bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
> > 
> > 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
> > 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
> > 
> > SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
> > 
> >       Recv     Send       Send                          Utilization       Service Demand
> >       Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
> >       Size     Size       Size    Time       Throughput local    remote   local   remote
> >       bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
> > 
> > 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
> > 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
> > 
> > The same tests were performed a multitude of time, and are always consistent (within a few percent).  I've also tried playing with various run-time kernel parameters (/proc/sys/kernel/net/...) on the 4.4.0 kernel to alleviate the issue but have had no success at all.
> > 
> > I'm at a loss as to what could possibly account for such a discrepancy...
> 
> Maybe new kernel is faster and you have drops somewhere ?
> 
> nstat >/dev/null
> netperf -H ...
> nstat
> 
> Would help
> 

Are you receiving my mails, or simply ignoring them ?

Thanks.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* RE: Poorer networking performance in later kernels?
  2016-04-18 12:16     ` Eric Dumazet
@ 2016-04-18 16:27       ` Butler, Peter
  2016-04-18 18:05         ` Eric Dumazet
  2016-04-18 16:44       ` Butler, Peter
  1 sibling, 1 reply; 16+ messages in thread
From: Butler, Peter @ 2016-04-18 16:27 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: netdev@vger.kernel.org

Hi Eric

Thanks for your response.  My apologies for being late in getting back to you - I wasn't able to have access to the lab hardware on the weekend.

I performed your test as suggested - I've provided a side-by-side diff of the nstat output below for the SCTP test only (not the TCP test).  Note that the fields that are output are somewhat different for the two kernels - i.e. some fields exist in one but not the other (presumably this comes from the kernel internals?).

Other than seeing 'larger' throughput numbers in this output I'm not sure what to take from it - I'm certainly not a networking expert :-(   Let me know if there's anything that speaks to you.

Note that this test was again done on a clean, freshly rebooted and idle system.  Let me know if there's any issues with the output format of this data in the email.

Thanks,

Peter

                   3.4.2                                                   4.4.0
-------------------------------------------------------------------------------------------------------
IpInReceives                 3457295         0.0   |   IpInReceives                 1151189         0.0
IpInDelivers                 3457295         0.0   |   IpInDelivers                 1151189         0.0
IpOutRequests                6864955         0.0   |   IpOutRequests                2249622         0.0
IcmpInErrors                 158             0.0   |   IcmpInErrors                 159             0.0
IcmpInTimeExcds              152             0.0   |   IcmpInTimeExcds              151             0.0
IcmpInEchoReps               6               0.0   |   IcmpInEchoReps               8               0.0
IcmpOutErrors                158             0.0   |   IcmpOutErrors                159             0.0
IcmpOutTimeExcds             152             0.0   |   IcmpOutTimeExcds             151             0.0
IcmpOutTimestamps            6               0.0   |   IcmpOutTimestamps            8               0.0
IcmpMsgInType3               152             0.0   |   IcmpMsgInType3               151             0.0
IcmpMsgInType8               6               0.0   |   IcmpMsgInType8               8               0.0
IcmpMsgOutType0              6               0.0   |   IcmpMsgOutType0              8               0.0
IcmpMsgOutType3              152             0.0   |   IcmpMsgOutType3              151             0.0
TcpActiveOpens               1               0.0       TcpActiveOpens               1               0.0
TcpPassiveOpens              3               0.0   |   TcpPassiveOpens              4               0.0
TcpInSegs                    70              0.0   |   TcpInSegs                    117             0.0
TcpOutSegs                   66              0.0   |   TcpOutSegs                   110             0.0
                                                   |   TcpOutRsts                   24              0.0
UdpInDatagrams               608             0.0   |   UdpInDatagrams               604             0.0
UdpNoPorts                   152             0.0   |   UdpNoPorts                   151             0.0
UdpOutDatagrams              760             0.0   |   UdpOutDatagrams              755             0.0
                                                   |   UdpIgnoredMulti              144             0.0
TcpExtTW                     2               0.0
TcpExtDelayedACKs            3               0.0   |   TcpExtDelayedACKs            4               0.0
TcpExtTCPHPHits              25              0.0   |   TcpExtTCPHPHits              41              0.0
TcpExtTCPPureAcks            12              0.0   |   TcpExtTCPPureAcks            14              0.0
TcpExtTCPHPAcks              18              0.0   |   TcpExtTCPHPAcks              26              0.0
                                                   |   TcpExtTCPRcvCoalesce         12              0.0
                                                   |   TcpExtTCPOrigDataSent        57              0.0
IpExtInBcastPkts             152             0.0   |   IpExtInBcastPkts             144             0.0
IpExtInOctets                166191161       0.0   |   IpExtInOctets                55395212        0.0
IpExtOutOctets               9107586685      0.0   |   IpExtOutOctets               2983660504      0.0
IpExtInBcastOctets           37356           0.0   |   IpExtInBcastOctets           35328           0.0
                                                   |   IpExtInNoECTPkts             1175            0.0
                                                   |   IpExtInECT0Pkts              1150014         0.0



-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com] 
Sent: April-18-16 8:17 AM
To: Butler, Peter <pbutler@sonusnet.com>
Cc: netdev@vger.kernel.org
Subject: Re: Poorer networking performance in later kernels?

On Fri, 2016-04-15 at 15:33 -0700, Eric Dumazet wrote:
> On Fri, 2016-04-15 at 21:02 +0000, Butler, Peter wrote:
> > (Please keep me CC'd to all comments/responses)
> > 
> > I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop in networking performance.  Nothing was changed on the test systems, other than the kernel itself (and kernel modules).  The identical .config used to build the 3.4.2 kernel was brought over into the 4.4.0 kernel source tree, and any configuration differences (e.g. new parameters, etc.) were taken as default values.
> > 
> > The testing was performed on the same actual hardware for both kernel versions (i.e. take the existing 3.4.2 physical setup, simply boot into the (new) kernel and run the same test).  The netperf utility was used for benchmarking and the testing was always performed on idle systems.
> > 
> > TCP testing yielded the following results, where the 4.4.0 kernel only got about 1/2 of the throughput:
> > 
> >       Recv     Send       Send                          Utilization       Service Demand
> >       Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
> >       Size     Size       Size    Time       Throughput local    remote   local   remote
> >       bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
> > 
> > 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
> > 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
> > 
> > SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
> > 
> >       Recv     Send       Send                          Utilization       Service Demand
> >       Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
> >       Size     Size       Size    Time       Throughput local    remote   local   remote
> >       bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
> > 
> > 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
> > 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
> > 
> > The same tests were performed a multitude of time, and are always consistent (within a few percent).  I've also tried playing with various run-time kernel parameters (/proc/sys/kernel/net/...) on the 4.4.0 kernel to alleviate the issue but have had no success at all.
> > 
> > I'm at a loss as to what could possibly account for such a discrepancy...
> 
> Maybe new kernel is faster and you have drops somewhere ?
> 
> nstat >/dev/null
> netperf -H ...
> nstat
> 
> Would help
> 

Are you receiving my mails, or simply ignoring them ?

Thanks.




^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Poorer networking performance in later kernels?
  2016-04-18 16:27       ` Butler, Peter
@ 2016-04-18 18:05         ` Eric Dumazet
  0 siblings, 0 replies; 16+ messages in thread
From: Eric Dumazet @ 2016-04-18 18:05 UTC (permalink / raw)
  To: Butler, Peter; +Cc: netdev@vger.kernel.org

On Mon, 2016-04-18 at 16:27 +0000, Butler, Peter wrote:
> Hi Eric
> 
> Thanks for your response.  My apologies for being late in getting back
> to you - I wasn't able to have access to the lab hardware on the
> weekend.
> 
> I performed your test as suggested - I've provided a side-by-side diff
> of the nstat output below for the SCTP test only (not the TCP test).
> Note that the fields that are output are somewhat different for the
> two kernels - i.e. some fields exist in one but not the other
> (presumably this comes from the kernel internals?).
> 
> Other than seeing 'larger' throughput numbers in this output I'm not
> sure what to take from it - I'm certainly not a networking
> expert :-(   Let me know if there's anything that speaks to you.
> 
> Note that this test was again done on a clean, freshly rebooted and
> idle system.  Let me know if there's any issues with the output format
> of this data in the email.
> 
> Thanks,
> 
> Peter


OK, please do not top-post on netdev.


I can not really comment on SCTP, could you please post numbers with
TCP ?

Thanks !

^ permalink raw reply	[flat|nested] 16+ messages in thread

* RE: Poorer networking performance in later kernels?
  2016-04-18 12:16     ` Eric Dumazet
  2016-04-18 16:27       ` Butler, Peter
@ 2016-04-18 16:44       ` Butler, Peter
  1 sibling, 0 replies; 16+ messages in thread
From: Butler, Peter @ 2016-04-18 16:44 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: netdev@vger.kernel.org

P.S.  Cancel my comment about some fields existing in one kernel but not the other- that was probably just an artefact of the fact that for one kernel the value was zero but not in the other kernel (and that I did not run nstat with the -z option to output zero counts).  So the data I provided is still good, but the 'empty' fields that exist in one kernel or the other can safely be assumed to be zero counts where left out.



-----Original Message-----
From: Butler, Peter 
Sent: April-18-16 12:27 PM
To: 'Eric Dumazet' <eric.dumazet@gmail.com>
Cc: netdev@vger.kernel.org
Subject: RE: Poorer networking performance in later kernels?

Hi Eric

Thanks for your response.  My apologies for being late in getting back to you - I wasn't able to have access to the lab hardware on the weekend.

I performed your test as suggested - I've provided a side-by-side diff of the nstat output below for the SCTP test only (not the TCP test).  Note that the fields that are output are somewhat different for the two kernels - i.e. some fields exist in one but not the other (presumably this comes from the kernel internals?).

Other than seeing 'larger' throughput numbers in this output I'm not sure what to take from it - I'm certainly not a networking expert :-(   Let me know if there's anything that speaks to you.

Note that this test was again done on a clean, freshly rebooted and idle system.  Let me know if there's any issues with the output format of this data in the email.

Thanks,

Peter

                   3.4.2                                                   4.4.0
-------------------------------------------------------------------------------------------------------
IpInReceives                 3457295         0.0   |   IpInReceives                 1151189         0.0
IpInDelivers                 3457295         0.0   |   IpInDelivers                 1151189         0.0
IpOutRequests                6864955         0.0   |   IpOutRequests                2249622         0.0
IcmpInErrors                 158             0.0   |   IcmpInErrors                 159             0.0
IcmpInTimeExcds              152             0.0   |   IcmpInTimeExcds              151             0.0
IcmpInEchoReps               6               0.0   |   IcmpInEchoReps               8               0.0
IcmpOutErrors                158             0.0   |   IcmpOutErrors                159             0.0
IcmpOutTimeExcds             152             0.0   |   IcmpOutTimeExcds             151             0.0
IcmpOutTimestamps            6               0.0   |   IcmpOutTimestamps            8               0.0
IcmpMsgInType3               152             0.0   |   IcmpMsgInType3               151             0.0
IcmpMsgInType8               6               0.0   |   IcmpMsgInType8               8               0.0
IcmpMsgOutType0              6               0.0   |   IcmpMsgOutType0              8               0.0
IcmpMsgOutType3              152             0.0   |   IcmpMsgOutType3              151             0.0
TcpActiveOpens               1               0.0       TcpActiveOpens               1               0.0
TcpPassiveOpens              3               0.0   |   TcpPassiveOpens              4               0.0
TcpInSegs                    70              0.0   |   TcpInSegs                    117             0.0
TcpOutSegs                   66              0.0   |   TcpOutSegs                   110             0.0
                                                   |   TcpOutRsts                   24              0.0
UdpInDatagrams               608             0.0   |   UdpInDatagrams               604             0.0
UdpNoPorts                   152             0.0   |   UdpNoPorts                   151             0.0
UdpOutDatagrams              760             0.0   |   UdpOutDatagrams              755             0.0
                                                   |   UdpIgnoredMulti              144             0.0
TcpExtTW                     2               0.0
TcpExtDelayedACKs            3               0.0   |   TcpExtDelayedACKs            4               0.0
TcpExtTCPHPHits              25              0.0   |   TcpExtTCPHPHits              41              0.0
TcpExtTCPPureAcks            12              0.0   |   TcpExtTCPPureAcks            14              0.0
TcpExtTCPHPAcks              18              0.0   |   TcpExtTCPHPAcks              26              0.0
                                                   |   TcpExtTCPRcvCoalesce         12              0.0
                                                   |   TcpExtTCPOrigDataSent        57              0.0
IpExtInBcastPkts             152             0.0   |   IpExtInBcastPkts             144             0.0
IpExtInOctets                166191161       0.0   |   IpExtInOctets                55395212        0.0
IpExtOutOctets               9107586685      0.0   |   IpExtOutOctets               2983660504      0.0
IpExtInBcastOctets           37356           0.0   |   IpExtInBcastOctets           35328           0.0
                                                   |   IpExtInNoECTPkts             1175            0.0
                                                   |   IpExtInECT0Pkts              1150014         0.0



-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com] 
Sent: April-18-16 8:17 AM
To: Butler, Peter <pbutler@sonusnet.com>
Cc: netdev@vger.kernel.org
Subject: Re: Poorer networking performance in later kernels?

On Fri, 2016-04-15 at 15:33 -0700, Eric Dumazet wrote:
> On Fri, 2016-04-15 at 21:02 +0000, Butler, Peter wrote:
> > (Please keep me CC'd to all comments/responses)
> > 
> > I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop in networking performance.  Nothing was changed on the test systems, other than the kernel itself (and kernel modules).  The identical .config used to build the 3.4.2 kernel was brought over into the 4.4.0 kernel source tree, and any configuration differences (e.g. new parameters, etc.) were taken as default values.
> > 
> > The testing was performed on the same actual hardware for both kernel versions (i.e. take the existing 3.4.2 physical setup, simply boot into the (new) kernel and run the same test).  The netperf utility was used for benchmarking and the testing was always performed on idle systems.
> > 
> > TCP testing yielded the following results, where the 4.4.0 kernel only got about 1/2 of the throughput:
> > 
> >       Recv     Send       Send                          Utilization       Service Demand
> >       Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
> >       Size     Size       Size    Time       Throughput local    remote   local   remote
> >       bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
> > 
> > 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
> > 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
> > 
> > SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
> > 
> >       Recv     Send       Send                          Utilization       Service Demand
> >       Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
> >       Size     Size       Size    Time       Throughput local    remote   local   remote
> >       bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
> > 
> > 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
> > 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
> > 
> > The same tests were performed a multitude of time, and are always consistent (within a few percent).  I've also tried playing with various run-time kernel parameters (/proc/sys/kernel/net/...) on the 4.4.0 kernel to alleviate the issue but have had no success at all.
> > 
> > I'm at a loss as to what could possibly account for such a discrepancy...
> 
> Maybe new kernel is faster and you have drops somewhere ?
> 
> nstat >/dev/null
> netperf -H ...
> nstat
> 
> Would help
> 

Are you receiving my mails, or simply ignoring them ?

Thanks.




^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Poorer networking performance in later kernels?
  2016-04-15 21:02 ` Poorer networking performance in later kernels? Butler, Peter
  2016-04-15 22:33   ` Eric Dumazet
@ 2016-04-15 22:37   ` Rick Jones
  2016-04-18 11:27     ` Butler, Peter
  2016-04-18 12:02     ` Butler, Peter
  1 sibling, 2 replies; 16+ messages in thread
From: Rick Jones @ 2016-04-15 22:37 UTC (permalink / raw)
  To: Butler, Peter, netdev@vger.kernel.org

On 04/15/2016 02:02 PM, Butler, Peter wrote:
> (Please keep me CC'd to all comments/responses)
>
> I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop
> in networking performance.  Nothing was changed on the test systems,
> other than the kernel itself (and kernel modules).  The identical
> .config used to build the 3.4.2 kernel was brought over into the
> 4.4.0 kernel source tree, and any configuration differences (e.g. new
> parameters, etc.) were taken as default values.
>
> The testing was performed on the same actual hardware for both kernel
> versions (i.e. take the existing 3.4.2 physical setup, simply boot
> into the (new) kernel and run the same test).  The netperf utility
> was used for benchmarking and the testing was always performed on
> idle systems.
>
> TCP testing yielded the following results, where the 4.4.0 kernel
> only got about 1/2 of the throughput:
>

>        Recv     Send       Send                          Utilization       Service Demand
>        Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>        Size     Size       Size    Time       Throughput local    remote   local   remote
>        bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
>
> 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
> 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
>
> SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
>
>        Recv     Send       Send                          Utilization       Service Demand
>        Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>        Size     Size       Size    Time       Throughput local    remote   local   remote
>        bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
>
> 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
> 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
>
> The same tests were performed a multitude of time, and are always
> consistent (within a few percent).  I've also tried playing with
> various run-time kernel parameters (/proc/sys/kernel/net/...) on the
> 4.4.0 kernel to alleviate the issue but have had no success at all.
>
> I'm at a loss as to what could possibly account for such a discrepancy...
>

I suspect I am not alone in being curious about the CPU(s) present in 
the systems and the model/whatnot of the NIC being used.  I'm also 
curious as to why you have what at first glance seem like absurdly large 
socket buffer sizes.

That said, it looks like you have some Really Big (tm) increases in 
service demand.  Many more CPU cycles being consumed per KB of data 
transferred.

Your message size makes me wonder if you were using a 9000 byte MTU.

Perhaps in the move from 3.4.2 to 4.4.0 you lost some or all of the 
stateless offloads for your NIC(s)?  Running ethtool -k <interface> on 
both ends under both kernels might be good.

Also, if you did have a 9000 byte MTU under 3.4.2 are you certain you 
still had it under 4.4.0?

It would (at least to me) also be interesting to run a TCP_RR test 
comparing the two kernels.  TCP_RR (at least with the default 
request/response size of one byte) doesn't really care about stateless 
offloads or MTUs and could show how much difference there is in basic 
path length (or I suppose in interrupt coalescing behaviour if the NIC 
in question has a mildly dodgy heuristic for such things).

happy benchmarking,

rick jones

^ permalink raw reply	[flat|nested] 16+ messages in thread

* RE: Poorer networking performance in later kernels?
  2016-04-15 22:37   ` Rick Jones
@ 2016-04-18 11:27     ` Butler, Peter
  2016-04-18 18:22       ` Rick Jones
  2016-04-18 12:02     ` Butler, Peter
  1 sibling, 1 reply; 16+ messages in thread
From: Butler, Peter @ 2016-04-18 11:27 UTC (permalink / raw)
  To: Rick Jones, netdev@vger.kernel.org

Hi Rick

Thanks for the reply.

Here is some hardware information, as requested (the two systems are identical, and are communicating with one another over a 10GB full-duplex Ethernet backplane):

- processor type: Intel(R) Xeon(R) CPU C5528  @ 2.13GHz
- NIC: Intel 82599EB 10GB XAUI/BX4
- NIC driver: ixgbe version 4.2.1-k (part of 4.4.0 kernel)

As for the buffer sizes, those rather large ones work fine for us with the 3.4.2 kernel.  However, for the sake of being complete, I have re-tried the tests with the 'standard' 4.4.0 kernel parameters for all /proc/sys/net/* values, and the results still were extremely poor in comparison to the 3.4.2 kernel.

Our MTU is actually just the standard 1500 bytes, however the message size was chosen to mimic actual traffic which will be segmented.

I ran ethtool -k (indeed I checked all ethtool parameters, not just those via -k) and the only real difference I could find was in "large-receive-offload" which was ON in 3.4.2 but OFF in 4.4.0 - so I used ethtool to change this to match the 3.4.2 settings and re-ran the tests.  Didn't help :-(   It's possible of course that I have missed a parameter here or there in comparing the 3.4.2 setup to the 4.4.0 setup.  I also tried running the ethtool config with the latest and greatest ethtool version (4.5) on the 4.4.0 kernel, as compared to the old 3.1 version on our 3.4.2 kernel.

I performed the TCP_RR test as requested and in that case, the results are much more comparable.  The old kernel is still better, but now only around 10% better as opposed to 2-3x better.

However I still contend that the *_STREAM tests are giving us more pertinent data, since our product application is only getting 1/3 to 1/2 half of the performance on the 4.4.0 kernel, and this is the same thing I see when I use netperf to test.

One other note: I tried running our 3.4.2 and 4.4.0 kernels in a VM environment on my workstation, so as to take the 'real' production hardware out of the equation.  When I perform the tests in this setup the 3.4.2 and 4.4.0 kernels perform identically - just as you would expect.

Any other ideas?  What can I be missing here?

Peter

-----Original Message-----
From: Rick Jones [mailto:rick.jones2@hpe.com] 
Sent: April-15-16 6:37 PM
To: Butler, Peter <pbutler@sonusnet.com>; netdev@vger.kernel.org
Subject: Re: Poorer networking performance in later kernels?

On 04/15/2016 02:02 PM, Butler, Peter wrote:
> (Please keep me CC'd to all comments/responses)
>
> I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop 
> in networking performance.  Nothing was changed on the test systems, 
> other than the kernel itself (and kernel modules).  The identical 
> .config used to build the 3.4.2 kernel was brought over into the
> 4.4.0 kernel source tree, and any configuration differences (e.g. new 
> parameters, etc.) were taken as default values.
>
> The testing was performed on the same actual hardware for both kernel 
> versions (i.e. take the existing 3.4.2 physical setup, simply boot 
> into the (new) kernel and run the same test).  The netperf utility was 
> used for benchmarking and the testing was always performed on idle 
> systems.
>
> TCP testing yielded the following results, where the 4.4.0 kernel only 
> got about 1/2 of the throughput:
>

>        Recv     Send       Send                          Utilization       Service Demand
>        Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>        Size     Size       Size    Time       Throughput local    remote   local   remote
>        bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
>
> 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
> 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
>
> SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
>
>        Recv     Send       Send                          Utilization       Service Demand
>        Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>        Size     Size       Size    Time       Throughput local    remote   local   remote
>        bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
>
> 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
> 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
>
> The same tests were performed a multitude of time, and are always 
> consistent (within a few percent).  I've also tried playing with 
> various run-time kernel parameters (/proc/sys/kernel/net/...) on the
> 4.4.0 kernel to alleviate the issue but have had no success at all.
>
> I'm at a loss as to what could possibly account for such a discrepancy...
>

I suspect I am not alone in being curious about the CPU(s) present in the systems and the model/whatnot of the NIC being used.  I'm also curious as to why you have what at first glance seem like absurdly large socket buffer sizes.

That said, it looks like you have some Really Big (tm) increases in service demand.  Many more CPU cycles being consumed per KB of data transferred.

Your message size makes me wonder if you were using a 9000 byte MTU.

Perhaps in the move from 3.4.2 to 4.4.0 you lost some or all of the stateless offloads for your NIC(s)?  Running ethtool -k <interface> on both ends under both kernels might be good.

Also, if you did have a 9000 byte MTU under 3.4.2 are you certain you still had it under 4.4.0?

It would (at least to me) also be interesting to run a TCP_RR test comparing the two kernels.  TCP_RR (at least with the default request/response size of one byte) doesn't really care about stateless offloads or MTUs and could show how much difference there is in basic path length (or I suppose in interrupt coalescing behaviour if the NIC in question has a mildly dodgy heuristic for such things).

happy benchmarking,

rick jones

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Poorer networking performance in later kernels?
  2016-04-18 11:27     ` Butler, Peter
@ 2016-04-18 18:22       ` Rick Jones
  2016-04-19 14:54         ` Butler, Peter
  0 siblings, 1 reply; 16+ messages in thread
From: Rick Jones @ 2016-04-18 18:22 UTC (permalink / raw)
  To: Butler, Peter, netdev@vger.kernel.org

On 04/18/2016 04:27 AM, Butler, Peter wrote:
> Hi Rick
>
> Thanks for the reply.
>
> Here is some hardware information, as requested (the two systems are
> identical, and are communicating with one another over a 10GB
> full-duplex Ethernet backplane):
>
> - processor type: Intel(R) Xeon(R) CPU C5528  @ 2.13GHz
> - NIC: Intel 82599EB 10GB XAUI/BX4
> - NIC driver: ixgbe version 4.2.1-k (part of 4.4.0 kernel)
>
> As for the buffer sizes, those rather large ones work fine for us
> with the 3.4.2 kernel.  However, for the sake of being complete, I
> have re-tried the tests with the 'standard' 4.4.0 kernel parameters
> for all /proc/sys/net/* values, and the results still were extremely
> poor in comparison to the 3.4.2 kernel.
>
> Our MTU is actually just the standard 1500 bytes, however the message
> size was chosen to mimic actual traffic which will be segmented.
>
> I ran ethtool -k (indeed I checked all ethtool parameters, not just
> those via -k) and the only real difference I could find was in
> "large-receive-offload" which was ON in 3.4.2 but OFF in 4.4.0 - so I
> used ethtool to change this to match the 3.4.2 settings and re-ran
> the tests.  Didn't help :-(   It's possible of course that I have
> missed a parameter here or there in comparing the 3.4.2 setup to the
> 4.4.0 setup.  I also tried running the ethtool config with the latest
> and greatest ethtool version (4.5) on the 4.4.0 kernel, as compared
> to the old 3.1 version on our 3.4.2 kernel.

So it would seem the stateless offloads are still enabled.  My next 
question would be to wonder if they are still "effective."  To that end, 
you could run a netperf test specifying a particular port number in the 
test-specific portion:

netperf ...   -- -P ,12345

and while that is running something like

tcpdump -s 96 -c 200000 -w /tmp/foo.pcap -i <interface> port 12345

then post-processed with the likes of:

tcpdump -n -r /tmp/foo.pcap | grep -v "length 0" | awk '{sum += 
$NF}END{print "average",sum/NR}'

the intent behind that is to see what the average post-GRO segment size 
happens to be on the receiver and then to compare it between the two 
kernels.  Grepping-away the "length 0" is to avoid counting ACKs and 
look only at data segments.  The specific port number is to avoid 
including any other connections which might happen to have traffic 
passing through at the time.

You could I suspect do the same comparison on the sending side.

There might I suppose be an easier way to get the average segment size - 
perhaps something from looking at ethtool stats - but the stone knives 
and bear skins of tcpdump above would have the added benefit of having a 
packet trace or three for someone to look at if they felt the need.  And 
for that, I would actually suggest starting the capture *before* the 
netperf test so the connection establishment is included.

> I performed the TCP_RR test as requested and in that case, the
> results are much more comparable.  The old kernel is still better,
> but now only around 10% better as opposed to 2-3x better.

Did the service demand change by 10% or just the transaction rate?

> However I still contend that the *_STREAM tests are giving us more
> pertinent data, since our product application is only getting 1/3 to
> 1/2 half of the performance on the 4.4.0 kernel, and this is the same
> thing I see when I use netperf to test.
>
> One other note: I tried running our 3.4.2 and 4.4.0 kernels in a VM
> environment on my workstation, so as to take the 'real' production
> hardware out of the equation.  When I perform the tests in this setup
> the 3.4.2 and 4.4.0 kernels perform identically - just as you would
> expect.

Running in a VM will likely change things massively and could I suppose 
mask other behaviour changes.

happy benchmarking,

rick jones
raj@tardy:~$ cat signatures/toppost
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

:)

>
> Any other ideas?  What can I be missing here?
>
> Peter
>
>
>
>
> -----Original Message-----
> From: Rick Jones [mailto:rick.jones2@hpe.com]
> Sent: April-15-16 6:37 PM
> To: Butler, Peter <pbutler@sonusnet.com>; netdev@vger.kernel.org
> Subject: Re: Poorer networking performance in later kernels?
>
> On 04/15/2016 02:02 PM, Butler, Peter wrote:
>> (Please keep me CC'd to all comments/responses)
>>
>> I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop
>> in networking performance.  Nothing was changed on the test systems,
>> other than the kernel itself (and kernel modules).  The identical
>> .config used to build the 3.4.2 kernel was brought over into the
>> 4.4.0 kernel source tree, and any configuration differences (e.g. new
>> parameters, etc.) were taken as default values.
>>
>> The testing was performed on the same actual hardware for both kernel
>> versions (i.e. take the existing 3.4.2 physical setup, simply boot
>> into the (new) kernel and run the same test).  The netperf utility was
>> used for benchmarking and the testing was always performed on idle
>> systems.
>>
>> TCP testing yielded the following results, where the 4.4.0 kernel only
>> got about 1/2 of the throughput:
>>
>
>>         Recv     Send       Send                          Utilization       Service Demand
>>         Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>>         Size     Size       Size    Time       Throughput local    remote   local   remote
>>         bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
>>
>> 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
>> 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
>>
>> SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
>>
>>         Recv     Send       Send                          Utilization       Service Demand
>>         Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>>         Size     Size       Size    Time       Throughput local    remote   local   remote
>>         bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
>>
>> 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
>> 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
>>
>> The same tests were performed a multitude of time, and are always
>> consistent (within a few percent).  I've also tried playing with
>> various run-time kernel parameters (/proc/sys/kernel/net/...) on the
>> 4.4.0 kernel to alleviate the issue but have had no success at all.
>>
>> I'm at a loss as to what could possibly account for such a discrepancy...
>>
>
> I suspect I am not alone in being curious about the CPU(s) present in the systems and the model/whatnot of the NIC being used.  I'm also curious as to why you have what at first glance seem like absurdly large socket buffer sizes.
>
> That said, it looks like you have some Really Big (tm) increases in service demand.  Many more CPU cycles being consumed per KB of data transferred.
>
> Your message size makes me wonder if you were using a 9000 byte MTU.
>
> Perhaps in the move from 3.4.2 to 4.4.0 you lost some or all of the stateless offloads for your NIC(s)?  Running ethtool -k <interface> on both ends under both kernels might be good.
>
> Also, if you did have a 9000 byte MTU under 3.4.2 are you certain you still had it under 4.4.0?
>
> It would (at least to me) also be interesting to run a TCP_RR test comparing the two kernels.  TCP_RR (at least with the default request/response size of one byte) doesn't really care about stateless offloads or MTUs and could show how much difference there is in basic path length (or I suppose in interrupt coalescing behaviour if the NIC in question has a mildly dodgy heuristic for such things).
>
> happy benchmarking,
>
> rick jones
>

^ permalink raw reply	[flat|nested] 16+ messages in thread

* RE: Poorer networking performance in later kernels?
  2016-04-18 18:22       ` Rick Jones
@ 2016-04-19 14:54         ` Butler, Peter
  2016-04-19 15:13           ` Josh Hunt
  2016-04-19 15:58           ` Oliver Hartkopp
  0 siblings, 2 replies; 16+ messages in thread
From: Butler, Peter @ 2016-04-19 14:54 UTC (permalink / raw)
  To: Rick Jones, netdev@vger.kernel.org

> -----Original Message-----
> From: Rick Jones [mailto:rick.jones2@hpe.com]
> Sent: April-15-16 6:37 PM
> To: Butler, Peter <pbutler@sonusnet.com>; netdev@vger.kernel.org
> Subject: Re: Poorer networking performance in later kernels?
>
> On 04/15/2016 02:02 PM, Butler, Peter wrote:
>> (Please keep me CC'd to all comments/responses)
>>
>> I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop 
>> in networking performance.  Nothing was changed on the test systems, 
>> other than the kernel itself (and kernel modules).  The identical 
>> .config used to build the 3.4.2 kernel was brought over into the
>> 4.4.0 kernel source tree, and any configuration differences (e.g. new 
>> parameters, etc.) were taken as default values.
>>
>> The testing was performed on the same actual hardware for both kernel 
>> versions (i.e. take the existing 3.4.2 physical setup, simply boot 
>> into the (new) kernel and run the same test).  The netperf utility 
>> was used for benchmarking and the testing was always performed on 
>> idle systems.
>>
>> TCP testing yielded the following results, where the 4.4.0 kernel 
>> only got about 1/2 of the throughput:
>>
>
>>         Recv     Send       Send                          Utilization       Service Demand
>>         Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>>         Size     Size       Size    Time       Throughput local    remote   local   remote
>>         bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
>>
>> 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
>> 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
>>
>> SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
>>
>>         Recv     Send       Send                          Utilization       Service Demand
>>         Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>>         Size     Size       Size    Time       Throughput local    remote   local   remote
>>         bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
>>
>> 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
>> 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
>>
>> The same tests were performed a multitude of time, and are always 
>> consistent (within a few percent).  I've also tried playing with 
>> various run-time kernel parameters (/proc/sys/kernel/net/...) on the
>> 4.4.0 kernel to alleviate the issue but have had no success at all.
>>
>> I'm at a loss as to what could possibly account for such a discrepancy...
>>
>
> I suspect I am not alone in being curious about the CPU(s) present in the systems and the model/whatnot of the NIC being used.  I'm also curious as to why you have what at first glance seem like absurdly large socket buffer sizes.
>
> That said, it looks like you have some Really Big (tm) increases in service demand.  Many more CPU cycles being consumed per KB of data transferred.
>
> Your message size makes me wonder if you were using a 9000 byte MTU.
>
> Perhaps in the move from 3.4.2 to 4.4.0 you lost some or all of the stateless offloads for your NIC(s)?  Running ethtool -k <interface> on both ends under both kernels might be good.
>
> Also, if you did have a 9000 byte MTU under 3.4.2 are you certain you still had it under 4.4.0?
>
> It would (at least to me) also be interesting to run a TCP_RR test comparing the two kernels.  TCP_RR (at least with the default request/response size of one byte) doesn't really care about stateless offloads or MTUs and could show how much difference there is in basic path length (or I suppose in interrupt coalescing behaviour if the NIC in question has a mildly dodgy heuristic for such things).
>
> happy benchmarking,
>
> rick jones
>


I think the issue is resolved.  I had to recompile my 4.4.0 kernel with a few options pertaining to the Intel NIC which somehow (?) got left out or otherwise clobbered when I ported my 3.4.2 .config to the 4.4.0 kernel source tree.  With those changes now in I see essentially identical performance with the two kernels.  Sorry for any confusion and/or waste of time here.  My bad.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Poorer networking performance in later kernels?
  2016-04-19 14:54         ` Butler, Peter
@ 2016-04-19 15:13           ` Josh Hunt
  2016-04-19 15:47             ` Butler, Peter
  2016-04-19 15:58           ` Oliver Hartkopp
  1 sibling, 1 reply; 16+ messages in thread
From: Josh Hunt @ 2016-04-19 15:13 UTC (permalink / raw)
  To: Butler, Peter; +Cc: Rick Jones, netdev@vger.kernel.org

On Tue, Apr 19, 2016 at 9:54 AM, Butler, Peter <pbutler@sonusnet.com> wrote:
>> -----Original Message-----
>> From: Rick Jones [mailto:rick.jones2@hpe.com]
>> Sent: April-15-16 6:37 PM
>> To: Butler, Peter <pbutler@sonusnet.com>; netdev@vger.kernel.org
>> Subject: Re: Poorer networking performance in later kernels?
>>
>> On 04/15/2016 02:02 PM, Butler, Peter wrote:
>>> (Please keep me CC'd to all comments/responses)
>>>
>>> I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop
>>> in networking performance.  Nothing was changed on the test systems,
>>> other than the kernel itself (and kernel modules).  The identical
>>> .config used to build the 3.4.2 kernel was brought over into the
>>> 4.4.0 kernel source tree, and any configuration differences (e.g. new
>>> parameters, etc.) were taken as default values.
>>>
>>> The testing was performed on the same actual hardware for both kernel
>>> versions (i.e. take the existing 3.4.2 physical setup, simply boot
>>> into the (new) kernel and run the same test).  The netperf utility
>>> was used for benchmarking and the testing was always performed on
>>> idle systems.
>>>
>>> TCP testing yielded the following results, where the 4.4.0 kernel
>>> only got about 1/2 of the throughput:
>>>
>>
>>>         Recv     Send       Send                          Utilization       Service Demand
>>>         Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>>>         Size     Size       Size    Time       Throughput local    remote   local   remote
>>>         bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
>>>
>>> 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
>>> 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
>>>
>>> SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
>>>
>>>         Recv     Send       Send                          Utilization       Service Demand
>>>         Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>>>         Size     Size       Size    Time       Throughput local    remote   local   remote
>>>         bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
>>>
>>> 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
>>> 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
>>>
>>> The same tests were performed a multitude of time, and are always
>>> consistent (within a few percent).  I've also tried playing with
>>> various run-time kernel parameters (/proc/sys/kernel/net/...) on the
>>> 4.4.0 kernel to alleviate the issue but have had no success at all.
>>>
>>> I'm at a loss as to what could possibly account for such a discrepancy...
>>>
>>
>> I suspect I am not alone in being curious about the CPU(s) present in the systems and the model/whatnot of the NIC being used.  I'm also curious as to why you have what at first glance seem like absurdly large socket buffer sizes.
>>
>> That said, it looks like you have some Really Big (tm) increases in service demand.  Many more CPU cycles being consumed per KB of data transferred.
>>
>> Your message size makes me wonder if you were using a 9000 byte MTU.
>>
>> Perhaps in the move from 3.4.2 to 4.4.0 you lost some or all of the stateless offloads for your NIC(s)?  Running ethtool -k <interface> on both ends under both kernels might be good.
>>
>> Also, if you did have a 9000 byte MTU under 3.4.2 are you certain you still had it under 4.4.0?
>>
>> It would (at least to me) also be interesting to run a TCP_RR test comparing the two kernels.  TCP_RR (at least with the default request/response size of one byte) doesn't really care about stateless offloads or MTUs and could show how much difference there is in basic path length (or I suppose in interrupt coalescing behaviour if the NIC in question has a mildly dodgy heuristic for such things).
>>
>> happy benchmarking,
>>
>> rick jones
>>
>
>
> I think the issue is resolved.  I had to recompile my 4.4.0 kernel with a few options pertaining to the Intel NIC which somehow (?) got left out or otherwise clobbered when I ported my 3.4.2 .config to the 4.4.0 kernel source tree.  With those changes now in I see essentially identical performance with the two kernels.  Sorry for any confusion and/or waste of time here.  My bad.
>
>

Can you share which config options you enabled to get your performance back?

-- 
Josh

^ permalink raw reply	[flat|nested] 16+ messages in thread

* RE: Poorer networking performance in later kernels?
  2016-04-19 15:13           ` Josh Hunt
@ 2016-04-19 15:47             ` Butler, Peter
  0 siblings, 0 replies; 16+ messages in thread
From: Butler, Peter @ 2016-04-19 15:47 UTC (permalink / raw)
  To: Josh Hunt; +Cc: Rick Jones, netdev@vger.kernel.org

On Tue, Apr 19, 2016 at 9:54 AM, Butler, Peter <pbutler@sonusnet.com> wrote:
>> -----Original Message-----
>> From: Rick Jones [mailto:rick.jones2@hpe.com]
>> Sent: April-15-16 6:37 PM
>> To: Butler, Peter <pbutler@sonusnet.com>; netdev@vger.kernel.org
>> Subject: Re: Poorer networking performance in later kernels?
>>
>> On 04/15/2016 02:02 PM, Butler, Peter wrote:
>>> (Please keep me CC'd to all comments/responses)
>>>
>>> I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked 
>>> drop in networking performance.  Nothing was changed on the test 
>>> systems, other than the kernel itself (and kernel modules).  The 
>>> identical .config used to build the 3.4.2 kernel was brought over 
>>> into the
>>> 4.4.0 kernel source tree, and any configuration differences (e.g. 
>>> new parameters, etc.) were taken as default values.
>>>
>>> The testing was performed on the same actual hardware for both 
>>> kernel versions (i.e. take the existing 3.4.2 physical setup, simply 
>>> boot into the (new) kernel and run the same test).  The netperf 
>>> utility was used for benchmarking and the testing was always 
>>> performed on idle systems.
>>>
>>> TCP testing yielded the following results, where the 4.4.0 kernel 
>>> only got about 1/2 of the throughput:
>>>
>>
>>>         Recv     Send       Send                          Utilization       Service Demand
>>>         Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>>>         Size     Size       Size    Time       Throughput local    remote   local   remote
>>>         bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
>>>
>>> 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
>>> 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
>>>
>>> SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
>>>
>>>         Recv     Send       Send                          Utilization       Service Demand
>>>         Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>>>         Size     Size       Size    Time       Throughput local    remote   local   remote
>>>         bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
>>>
>>> 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
>>> 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
>>>
>>> The same tests were performed a multitude of time, and are always 
>>> consistent (within a few percent).  I've also tried playing with 
>>> various run-time kernel parameters (/proc/sys/kernel/net/...) on the
>>> 4.4.0 kernel to alleviate the issue but have had no success at all.
>>>
>>> I'm at a loss as to what could possibly account for such a discrepancy...
>>>
>>
>> I suspect I am not alone in being curious about the CPU(s) present in the systems and the model/whatnot of the NIC being used.  I'm also curious as to why you have what at first glance seem like absurdly large socket buffer sizes.
>>
>> That said, it looks like you have some Really Big (tm) increases in service demand.  Many more CPU cycles being consumed per KB of data transferred.
>>
>> Your message size makes me wonder if you were using a 9000 byte MTU.
>>
>> Perhaps in the move from 3.4.2 to 4.4.0 you lost some or all of the stateless offloads for your NIC(s)?  Running ethtool -k <interface> on both ends under both kernels might be good.
>>
>> Also, if you did have a 9000 byte MTU under 3.4.2 are you certain you still had it under 4.4.0?
>>
>> It would (at least to me) also be interesting to run a TCP_RR test comparing the two kernels.  TCP_RR (at least with the default request/response size of one byte) doesn't really care about stateless offloads or MTUs and could show how much difference there is in basic path length (or I suppose in interrupt coalescing behaviour if the NIC in question has a mildly dodgy heuristic for such things).
>>
>> happy benchmarking,
>>
>> rick jones
>>
>
>
> I think the issue is resolved.  I had to recompile my 4.4.0 kernel with a few options pertaining to the Intel NIC which somehow (?) got left out or otherwise clobbered when I ported my 3.4.2 .config to the 4.4.0 kernel source tree.  With those changes now in I see essentially identical performance with the two kernels.  Sorry for any confusion and/or waste of time here.  My bad.
>
>

Can you share which config options you enabled to get your performance back?

--
Josh


I should have been more specific.  What I actually did was (re)copy my old 3.4.2 config into the 4.4.0 tree and let the kernel build populate any new parameters with defaults.  So I can provide a complete diff of the two .config files, but the actual subset that accounts for the 'fix' would only be a few of these parameters such as those pertaining to the Intel/NIC stuff.  Sorry.    In any case, here's the diff - again, the actual culprits are probably only a few of the entries in this diff.

3c3
< # Linux/x86_64 3.4.2-1.fc16.x86_64 Kernel Configuration
---
> # Linux/x86 4.4.0 Kernel Configuration
6d5
< # CONFIG_X86_32 is not set
9a9
> CONFIG_PERF_EVENTS_INTEL_UNCORE=y
12,16d11
< CONFIG_GENERIC_CMOS_UPDATE=y
< CONFIG_CLOCKSOURCE_WATCHDOG=y
< CONFIG_GENERIC_CLOCKEVENTS=y
< CONFIG_ARCH_CLOCKSOURCE_DATA=y
< CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
28d22
< # CONFIG_RWSEM_GENERIC_SPINLOCK is not set
30d23
< CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
32d24
< CONFIG_GENERIC_TIME_VSYSCALL=y
34d25
< CONFIG_ARCH_HAS_DEFAULT_IDLE=y
36d26
< CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
41a32,33
> CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
> CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
48d39
< CONFIG_X86_HT=y
50,51c41,43
< # CONFIG_KTIME_SCALAR is not set
< CONFIG_ARCH_CPU_PROBE_RELEASE=y
---
> CONFIG_ARCH_SUPPORTS_UPROBES=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_PGTABLE_LEVELS=4
53d44
< CONFIG_HAVE_IRQ_WORK=y
54a46
> CONFIG_BUILDTIME_EXTABLE_SORT=y
59d50
< CONFIG_EXPERIMENTAL=y
61a53
> # CONFIG_COMPILE_TEST is not set
68a61
> CONFIG_HAVE_KERNEL_LZ4=y
73a67
> # CONFIG_KERNEL_LZ4 is not set
80,81c74
< CONFIG_BSD_PROCESS_ACCT=y
< CONFIG_BSD_PROCESS_ACCT_V3=y
---
> CONFIG_CROSS_MEMORY_ATTACH=y
83,86c76
< CONFIG_TASKSTATS=y
< CONFIG_TASK_DELAY_ACCT=y
< CONFIG_TASK_XACCT=y
< CONFIG_TASK_IO_ACCOUNTING=y
---
> CONFIG_USELIB=y
87a78
> CONFIG_HAVE_ARCH_AUDITSYSCALL=y
91,92d81
< # CONFIG_AUDIT_LOGINUID_IMMUTABLE is not set
< CONFIG_HAVE_GENERIC_HARDIRQS=y
97d85
< CONFIG_GENERIC_HARDIRQS=y
100a89,93
> CONFIG_IRQ_DOMAIN=y
> CONFIG_IRQ_DOMAIN_HIERARCHY=y
> CONFIG_GENERIC_MSI_IRQ=y
> CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
> # CONFIG_IRQ_DOMAIN_DEBUG is not set
102a96,127
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_ARCH_CLOCKSOURCE_DATA=y
> CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE=y
> CONFIG_GENERIC_TIME_VSYSCALL=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
> CONFIG_GENERIC_CMOS_UPDATE=y
> 
> #
> # Timers subsystem
> #
> CONFIG_TICK_ONESHOT=y
> CONFIG_NO_HZ_COMMON=y
> # CONFIG_HZ_PERIODIC is not set
> CONFIG_NO_HZ_IDLE=y
> # CONFIG_NO_HZ_FULL is not set
> CONFIG_NO_HZ=y
> CONFIG_HIGH_RES_TIMERS=y
> 
> #
> # CPU/Task time and stats accounting
> #
> # CONFIG_TICK_CPU_ACCOUNTING is not set
> # CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
> CONFIG_IRQ_TIME_ACCOUNTING=y
> CONFIG_BSD_PROCESS_ACCT=y
> CONFIG_BSD_PROCESS_ACCT_V3=y
> CONFIG_TASKSTATS=y
> CONFIG_TASK_DELAY_ACCT=y
> CONFIG_TASK_XACCT=y
> CONFIG_TASK_IO_ACCOUNTING=y
108,111c133,136
< # CONFIG_PREEMPT_RCU is not set
< CONFIG_RCU_FANOUT=64
< # CONFIG_RCU_FANOUT_EXACT is not set
< # CONFIG_RCU_FAST_NO_HZ is not set
---
> # CONFIG_RCU_EXPERT is not set
> CONFIG_SRCU=y
> # CONFIG_TASKS_RCU is not set
> CONFIG_RCU_STALL_COMMON=y
112a138,139
> # CONFIG_RCU_EXPEDITE_BOOT is not set
> # CONFIG_BUILD_BIN2C is not set
114a142
> CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
115a144,147
> CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
> CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
> CONFIG_ARCH_SUPPORTS_INT128=y
> # CONFIG_NUMA_BALANCING is not set
118a151
> # CONFIG_CGROUP_PIDS is not set
123,127c156,157
< CONFIG_RESOURCE_COUNTERS=y
< CONFIG_CGROUP_MEM_RES_CTLR=y
< CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
< # CONFIG_CGROUP_MEM_RES_CTLR_SWAP_ENABLED is not set
< CONFIG_CGROUP_MEM_RES_CTLR_KMEM=y
---
> # CONFIG_MEMCG is not set
> # CONFIG_CGROUP_HUGETLB is not set
143d172
< CONFIG_MM_OWNER=y
152a182
> CONFIG_RD_LZ4=y
155a186,189
> CONFIG_HAVE_UID16=y
> CONFIG_SYSCTL_EXCEPTION_TRACE=y
> CONFIG_HAVE_PCSPKR_PLATFORM=y
> CONFIG_BPF=y
157a192,194
> CONFIG_MULTIUSER=y
> CONFIG_SGETMASK_SYSCALL=y
> CONFIG_SYSFS_SYSCALL=y
161d197
< CONFIG_HOTPLUG=y
166d201
< CONFIG_HAVE_PCSPKR_PLATFORM=y
172a208
> # CONFIG_BPF_SYSCALL is not set
174a211,214
> CONFIG_ADVISE_SYSCALLS=y
> # CONFIG_USERFAULTFD is not set
> CONFIG_PCI_QUIRKS=y
> CONFIG_MEMBARRIER=y
182d221
< CONFIG_PERF_COUNTERS=y
185d223
< CONFIG_PCI_QUIRKS=y
189a228,229
> CONFIG_SLUB_CPU_PARTIAL=y
> # CONFIG_SYSTEM_DATA_VERIFICATION is not set
191a232
> CONFIG_KEXEC_CORE=y
197a239
> # CONFIG_STATIC_KEYS_SELFTEST is not set
198a241,243
> CONFIG_KPROBES_ON_FTRACE=y
> # CONFIG_UPROBES is not set
> # CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
199a245
> CONFIG_ARCH_USE_BUILTIN_BSWAP=y
205a252
> CONFIG_HAVE_KPROBES_ON_FTRACE=y
208c255,257
< CONFIG_USE_GENERIC_SMP_HELPERS=y
---
> CONFIG_HAVE_DMA_CONTIGUOUS=y
> CONFIG_GENERIC_SMP_IDLE_THREAD=y
> CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
214a264,265
> CONFIG_HAVE_PERF_REGS=y
> CONFIG_HAVE_PERF_USER_STACK_DUMP=y
219a271
> CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
220a273,291
> CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
> CONFIG_SECCOMP_FILTER=y
> CONFIG_HAVE_CC_STACKPROTECTOR=y
> # CONFIG_CC_STACKPROTECTOR is not set
> CONFIG_CC_STACKPROTECTOR_NONE=y
> # CONFIG_CC_STACKPROTECTOR_REGULAR is not set
> # CONFIG_CC_STACKPROTECTOR_STRONG is not set
> CONFIG_HAVE_CONTEXT_TRACKING=y
> CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
> CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
> CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
> CONFIG_HAVE_ARCH_HUGE_VMAP=y
> CONFIG_HAVE_ARCH_SOFT_DIRTY=y
> CONFIG_MODULES_USE_ELF_RELA=y
> CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
> CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
> CONFIG_HAVE_COPY_THREAD_TLS=y
> CONFIG_OLD_SIGSUSPEND3=y
> CONFIG_COMPAT_OLD_SIGACTION=y
225a297
> CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
236c308,310
< CONFIG_STOP_MACHINE=y
---
> # CONFIG_MODULE_SIG is not set
> # CONFIG_MODULE_COMPRESS is not set
> CONFIG_MODULES_TREE_LOOKUP=y
241a316
> # CONFIG_BLK_CMDLINE_PARSER is not set
247a323
> # CONFIG_AIX_PARTITION is not set
263a340
> # CONFIG_CMDLINE_PARTITION is not set
279,285d355
< # CONFIG_INLINE_SPIN_TRYLOCK is not set
< # CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
< # CONFIG_INLINE_SPIN_LOCK is not set
< # CONFIG_INLINE_SPIN_LOCK_BH is not set
< # CONFIG_INLINE_SPIN_LOCK_IRQ is not set
< # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
< # CONFIG_INLINE_SPIN_UNLOCK_BH is not set
287,292d356
< # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
< # CONFIG_INLINE_READ_TRYLOCK is not set
< # CONFIG_INLINE_READ_LOCK is not set
< # CONFIG_INLINE_READ_LOCK_BH is not set
< # CONFIG_INLINE_READ_LOCK_IRQ is not set
< # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
294d357
< # CONFIG_INLINE_READ_UNLOCK_BH is not set
296,301d358
< # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
< # CONFIG_INLINE_WRITE_TRYLOCK is not set
< # CONFIG_INLINE_WRITE_LOCK is not set
< # CONFIG_INLINE_WRITE_LOCK_BH is not set
< # CONFIG_INLINE_WRITE_LOCK_IRQ is not set
< # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
303d359
< # CONFIG_INLINE_WRITE_UNLOCK_BH is not set
305c361
< # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
---
> CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
306a363,368
> CONFIG_RWSEM_SPIN_ON_OWNER=y
> CONFIG_LOCK_SPIN_ON_OWNER=y
> CONFIG_ARCH_USE_QUEUED_SPINLOCKS=y
> CONFIG_QUEUED_SPINLOCKS=y
> CONFIG_ARCH_USE_QUEUED_RWLOCKS=y
> CONFIG_QUEUED_RWLOCKS=y
313,317d374
< CONFIG_TICK_ONESHOT=y
< CONFIG_NO_HZ=y
< CONFIG_HIGH_RES_TIMERS=y
< CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
< CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
318a376
> CONFIG_X86_FEATURE_NAMES=y
324a383,386
> # CONFIG_X86_GOLDFISH is not set
> # CONFIG_X86_INTEL_LPSS is not set
> # CONFIG_X86_AMD_PLATFORM_DEVICE is not set
> # CONFIG_IOSF_MBI is not set
327,341c389
< CONFIG_PARAVIRT_GUEST=y
< CONFIG_PARAVIRT_TIME_ACCOUNTING=y
< CONFIG_XEN=y
< CONFIG_XEN_DOM0=y
< CONFIG_XEN_PRIVILEGED_GUEST=y
< CONFIG_XEN_PVHVM=y
< CONFIG_XEN_MAX_DOMAIN_MEMORY=500
< CONFIG_XEN_SAVE_RESTORE=y
< CONFIG_XEN_DEBUG_FS=y
< CONFIG_KVM_CLOCK=y
< CONFIG_KVM_GUEST=y
< CONFIG_PARAVIRT=y
< # CONFIG_PARAVIRT_SPINLOCKS is not set
< CONFIG_PARAVIRT_CLOCK=y
< # CONFIG_PARAVIRT_DEBUG is not set
---
> # CONFIG_HYPERVISOR_GUEST is not set
343d390
< # CONFIG_MEMTEST is not set
350d396
< CONFIG_X86_CMPXCHG=y
352,353d397
< CONFIG_X86_XADD=y
< CONFIG_X86_WP_WORKS_OK=y
373d416
< CONFIG_IRQ_TIME_ACCOUNTING=y
385a429,432
> # CONFIG_VM86 is not set
> CONFIG_X86_16BIT=y
> CONFIG_X86_ESPFIX64=y
> CONFIG_X86_VSYSCALL_EMULATION=y
387c434
< CONFIG_MICROCODE=m
---
> CONFIG_MICROCODE=y
395c442
< CONFIG_DIRECT_GBPAGES=y
---
> CONFIG_X86_DIRECT_GBPAGES=y
418a466,468
> CONFIG_MEMORY_ISOLATION=y
> # CONFIG_MOVABLE_NODE is not set
> # CONFIG_HAVE_BOOTMEM_INFO_NODE is not set
420d469
< CONFIG_PAGEFLAGS_EXTENDED=y
421a471,473
> CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
> CONFIG_MEMORY_BALLOON=y
> CONFIG_BALLOON_COMPACTION=y
423a476
> CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
437a491,499
> # CONFIG_FRONTSWAP is not set
> # CONFIG_CMA is not set
> # CONFIG_ZPOOL is not set
> # CONFIG_ZBUD is not set
> # CONFIG_ZSMALLOC is not set
> CONFIG_GENERIC_EARLY_IOREMAP=y
> CONFIG_ARCH_SUPPORTS_DEFERRED_STRUCT_PAGE_INIT=y
> # CONFIG_IDLE_PAGE_TRACKING is not set
> # CONFIG_X86_PMEM_LEGACY is not set
447a510,511
> CONFIG_X86_SMAP=y
> # CONFIG_X86_INTEL_MPX is not set
449a514
> # CONFIG_EFI_MIXED is not set
451d515
< CONFIG_CC_STACKPROTECTOR=y
458a523
> # CONFIG_KEXEC_FILE is not set
462a528
> # CONFIG_RANDOMIZE_BASE is not set
464a531,532
> # CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set
> # CONFIG_DEBUG_HOTPLUG_CPU0 is not set
465a534,536
> # CONFIG_LEGACY_VSYSCALL_NATIVE is not set
> CONFIG_LEGACY_VSYSCALL_EMULATE=y
> # CONFIG_LEGACY_VSYSCALL_NONE is not set
466a538,540
> CONFIG_MODIFY_LDT_SYSCALL=y
> CONFIG_HAVE_LIVEPATCH=y
> # CONFIG_LIVEPATCH is not set
481c555,556
< CONFIG_PM_RUNTIME=y
---
> # CONFIG_PM_AUTOSLEEP is not set
> # CONFIG_PM_WAKELOCKS is not set
486c561,562
< CONFIG_CAN_PM_TRACE=y
---
> CONFIG_PM_SLEEP_DEBUG=y
> # CONFIG_DPM_WATCHDOG is not set
488a565
> # CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
489a567,570
> CONFIG_ACPI_LEGACY_TABLES_LOOKUP=y
> CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC=y
> CONFIG_ACPI_SYSTEM_POWER_STATES_SUPPORT=y
> # CONFIG_ACPI_DEBUGGER is not set
491d571
< CONFIG_ACPI_PROCFS=y
492a573
> CONFIG_ACPI_REV_OVERRIDE_POSSIBLE=y
494d574
< # CONFIG_ACPI_PROC_EVENT is not set
500a581,582
> CONFIG_ACPI_CPU_FREQ_PSS=y
> CONFIG_ACPI_PROCESSOR_IDLE=y
508c590
< CONFIG_ACPI_BLACKLIST_YEAR=0
---
> # CONFIG_ACPI_INITRD_TABLE_OVERRIDE is not set
512a595
> CONFIG_ACPI_HOTPLUG_IOAPIC=y
516c599,603
< CONFIG_ACPI_BGRT=m
---
> # CONFIG_ACPI_BGRT is not set
> # CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not set
> # CONFIG_ACPI_NFIT is not set
> CONFIG_HAVE_ACPI_APEI=y
> CONFIG_HAVE_ACPI_APEI_NMI=y
522a610,611
> # CONFIG_ACPI_EXTLOG is not set
> # CONFIG_PMIC_OPREGION is not set
529c618
< CONFIG_CPU_FREQ_TABLE=y
---
> CONFIG_CPU_FREQ_GOV_COMMON=y
532a622
> # CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
543c633
< # x86 CPU frequency scaling drivers
---
> # CPU frequency scaling drivers
544a635
> # CONFIG_X86_INTEL_PSTATE is not set
546a638
> CONFIG_X86_ACPI_CPUFREQ_CPB=y
547a640
> # CONFIG_X86_AMD_FREQ_SENSITIVITY is not set
554a648,651
> 
> #
> # CPU Idle
> #
557a655
> # CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
572d669
< CONFIG_PCI_XEN=y
574d670
< # CONFIG_PCI_CNB20LE_QUIRK is not set
586c682
< CONFIG_ARCH_SUPPORTS_MSI=y
---
> CONFIG_PCI_BUS_ADDR_T_64BIT=y
587a684
> CONFIG_PCI_MSI_IRQ_DOMAIN=y
591d687
< CONFIG_XEN_PCIDEV_FRONTEND=m
597d692
< CONFIG_PCI_IOAPIC=y
598a694,697
> 
> #
> # PCI host controller drivers
> #
619d717
< CONFIG_HOTPLUG_PCI_FAKE=m
624a723
> # CONFIG_X86_SYSFB is not set
631d729
< CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
632a731
> CONFIG_BINFMT_SCRIPT=y
634a734
> CONFIG_COREDUMP=y
642c742,743
< CONFIG_HAVE_TEXT_POKE_SMP=y
---
> CONFIG_X86_DEV_DMA_OPS=y
> CONFIG_PMC_ATOM=y
644a746
> CONFIG_NET_INGRESS=y
649a752
> # CONFIG_PACKET_DIAG is not set
652a756
> CONFIG_XFRM_ALGO=y
670a775
> CONFIG_NET_IP_TUNNEL=m
677d781
< CONFIG_ARPD=y
678a783,786
> # CONFIG_NET_IPVTI is not set
> CONFIG_NET_UDP_TUNNEL=m
> # CONFIG_NET_FOU is not set
> # CONFIG_NET_FOU_IP_TUNNELS is not set
703a812,813
> # CONFIG_TCP_CONG_DCTCP is not set
> # CONFIG_TCP_CONG_CDG is not set
709d818
< CONFIG_IPV6_PRIVACY=y
716a826
> # CONFIG_IPV6_ILA is not set
722a833
> # CONFIG_IPV6_VTI is not set
726a838
> # CONFIG_IPV6_GRE is not set
733a846
> CONFIG_NET_PTP_CLASSIFY=y
738c851
< CONFIG_BRIDGE_NETFILTER=y
---
> CONFIG_BRIDGE_NETFILTER=m
742a856
> CONFIG_NETFILTER_INGRESS=y
747a862
> CONFIG_NF_LOG_COMMON=m
772c887,888
< CONFIG_NETFILTER_TPROXY=m
---
> # CONFIG_NETFILTER_NETLINK_GLUE_CT is not set
> # CONFIG_NF_TABLES is not set
792a909
> # CONFIG_NETFILTER_XT_TARGET_HMARK is not set
811a929,930
> # CONFIG_NETFILTER_XT_MATCH_BPF is not set
> # CONFIG_NETFILTER_XT_MATCH_CGROUP is not set
814a934
> # CONFIG_NETFILTER_XT_MATCH_CONNLABEL is not set
826a947
> # CONFIG_NETFILTER_XT_MATCH_IPCOMP is not set
828a950
> CONFIG_NETFILTER_XT_MATCH_L2TP=m
857a980
> # CONFIG_IP_SET_HASH_IPMARK is not set
860a984,985
> # CONFIG_IP_SET_HASH_MAC is not set
> # CONFIG_IP_SET_HASH_NETPORTNET is not set
861a987
> # CONFIG_IP_SET_HASH_NETNET is not set
886a1013,1014
> # CONFIG_IP_VS_FO is not set
> # CONFIG_IP_VS_OVF is not set
902d1029
< CONFIG_IP_VS_FTP=m
912c1039,1043
< CONFIG_IP_NF_QUEUE=m
---
> CONFIG_NF_DUP_IPV4=m
> # CONFIG_NF_LOG_ARP is not set
> CONFIG_NF_LOG_IPV4=m
> CONFIG_NF_REJECT_IPV4=y
> # CONFIG_NF_NAT_IPV4 is not set
920,937c1051,1052
< CONFIG_IP_NF_TARGET_ULOG=m
< CONFIG_NF_NAT=m
< CONFIG_NF_NAT_NEEDED=y
< CONFIG_IP_NF_TARGET_MASQUERADE=m
< CONFIG_IP_NF_TARGET_NETMAP=m
< CONFIG_IP_NF_TARGET_REDIRECT=m
< CONFIG_NF_NAT_SNMP_BASIC=m
< CONFIG_NF_NAT_PROTO_DCCP=m
< CONFIG_NF_NAT_PROTO_GRE=m
< CONFIG_NF_NAT_PROTO_UDPLITE=m
< CONFIG_NF_NAT_PROTO_SCTP=m
< CONFIG_NF_NAT_FTP=m
< CONFIG_NF_NAT_IRC=m
< CONFIG_NF_NAT_TFTP=m
< CONFIG_NF_NAT_AMANDA=m
< CONFIG_NF_NAT_PPTP=m
< CONFIG_NF_NAT_H323=m
< CONFIG_NF_NAT_SIP=m
---
> # CONFIG_IP_NF_TARGET_SYNPROXY is not set
> # CONFIG_IP_NF_NAT is not set
953c1068,1071
< CONFIG_IP6_NF_QUEUE=m
---
> CONFIG_NF_DUP_IPV6=m
> CONFIG_NF_REJECT_IPV6=m
> CONFIG_NF_LOG_IPV6=m
> # CONFIG_NF_NAT_IPV6 is not set
966a1085
> # CONFIG_IP6_NF_TARGET_SYNPROXY is not set
969a1089
> # CONFIG_IP6_NF_NAT is not set
990d1109
< CONFIG_BRIDGE_EBT_ULOG=m
996c1115
< # DCCP CCIDs Configuration (EXPERIMENTAL)
---
> # DCCP CCIDs Configuration
1010d1128
< # CONFIG_SCTP_DBG_MSG is not set
1012,1014c1130,1134
< # CONFIG_SCTP_HMAC_NONE is not set
< CONFIG_SCTP_HMAC_SHA1=y
< # CONFIG_SCTP_HMAC_MD5 is not set
---
> CONFIG_SCTP_DEFAULT_COOKIE_HMAC_MD5=y
> # CONFIG_SCTP_DEFAULT_COOKIE_HMAC_SHA1 is not set
> # CONFIG_SCTP_DEFAULT_COOKIE_HMAC_NONE is not set
> CONFIG_SCTP_COOKIE_HMAC_MD5=y
> # CONFIG_SCTP_COOKIE_HMAC_SHA1 is not set
1020d1139
< CONFIG_TIPC_PORTS=8191
1021a1141
> CONFIG_TIPC_MEDIA_UDP=y
1038,1041c1158,1159
< CONFIG_NET_DSA=y
< CONFIG_NET_DSA_TAG_DSA=y
< CONFIG_NET_DSA_TAG_EDSA=y
< CONFIG_NET_DSA_TAG_TRAILER=y
---
> # CONFIG_BRIDGE_VLAN_FILTERING is not set
> CONFIG_HAVE_NET_DSA=y
1043a1162
> # CONFIG_VLAN_8021Q_MVRP is not set
1053d1171
< CONFIG_IPDDP_DECAP=y
1056,1057d1173
< # CONFIG_ECONET is not set
< CONFIG_WAN_ROUTER=m
1058a1175
> # CONFIG_6LOWPAN is not set
1060c1177,1179
< CONFIG_IEEE802154_6LOWPAN=m
---
> # CONFIG_IEEE802154_NL802154_EXPERIMENTAL is not set
> CONFIG_IEEE802154_SOCKET=m
> # CONFIG_MAC802154 is not set
1083a1203,1207
> # CONFIG_NET_SCH_CODEL is not set
> # CONFIG_NET_SCH_FQ_CODEL is not set
> # CONFIG_NET_SCH_FQ is not set
> # CONFIG_NET_SCH_HHF is not set
> # CONFIG_NET_SCH_PIE is not set
1101a1226,1227
> # CONFIG_NET_CLS_BPF is not set
> # CONFIG_NET_CLS_FLOWER is not set
1108a1235
> # CONFIG_NET_EMATCH_IPSET is not set
1119a1247,1249
> # CONFIG_NET_ACT_VLAN is not set
> # CONFIG_NET_ACT_BPF is not set
> # CONFIG_NET_ACT_CONNMARK is not set
1125a1256,1265
> CONFIG_OPENVSWITCH_GRE=m
> # CONFIG_VSOCKETS is not set
> # CONFIG_NETLINK_MMAP is not set
> # CONFIG_NETLINK_DIAG is not set
> CONFIG_MPLS=y
> CONFIG_NET_MPLS_GSO=m
> # CONFIG_MPLS_ROUTING is not set
> # CONFIG_HSR is not set
> # CONFIG_NET_SWITCHDEV is not set
> # CONFIG_NET_L3_MASTER_DEV is not set
1129c1269,1271
< CONFIG_NETPRIO_CGROUP=m
---
> # CONFIG_CGROUP_NET_PRIO is not set
> CONFIG_CGROUP_NET_CLASSID=y
> CONFIG_NET_RX_BUSY_POLL=y
1131d1272
< CONFIG_HAVE_BPF_JIT=y
1132a1274
> CONFIG_NET_FLOW_LIMIT=y
1217a1360
> CONFIG_BT_BREDR=y
1224a1368,1371
> CONFIG_BT_HS=y
> CONFIG_BT_LE=y
> # CONFIG_BT_SELFTEST is not set
> CONFIG_BT_DEBUGFS=y
1228a1376,1378
> CONFIG_BT_INTEL=m
> CONFIG_BT_BCM=m
> CONFIG_BT_RTL=m
1229a1380,1381
> CONFIG_BT_HCIBTUSB_BCM=y
> CONFIG_BT_HCIBTUSB_RTL=y
1235a1388,1391
> # CONFIG_BT_HCIUART_3WIRE is not set
> # CONFIG_BT_HCIUART_INTEL is not set
> # CONFIG_BT_HCIUART_BCM is not set
> # CONFIG_BT_HCIUART_QCA is not set
1261a1418
> CONFIG_CFG80211_CRDA_SUPPORT=y
1263c1420
< CONFIG_WIRELESS_EXT_SYSFS=y
---
> CONFIG_CFG80211_WEXT_EXPORT=y
1272a1430
> # CONFIG_MAC80211_RC_MINSTREL_VHT is not set
1277a1436
> # CONFIG_MAC80211_MESSAGE_TRACING is not set
1278a1438
> CONFIG_MAC80211_STA_HASH_MAX_SIZE=0
1292a1453,1454
> # CONFIG_LWTUNNEL is not set
> CONFIG_HAVE_BPF_JIT=y
1300a1463
> CONFIG_UEVENT_HELPER=y
1308a1472,1476
> CONFIG_FW_LOADER_USER_HELPER=y
> # CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set
> CONFIG_WANT_DEV_COREDUMP=y
> CONFIG_ALLOW_DEV_COREDUMP=y
> CONFIG_DEV_COREDUMP=y
1311c1479
< CONFIG_SYS_HYPERVISOR=y
---
> # CONFIG_SYS_HYPERVISOR is not set
1312a1481
> CONFIG_GENERIC_CPU_AUTOPROBE=y
1315a1485,1489
> # CONFIG_FENCE_TRACE is not set
> 
> #
> # Bus devices
> #
1323a1498
> # CONFIG_MTD_CMDLINE_PARTS is not set
1329d1503
< CONFIG_MTD_CHAR=m
1341a1516
> # CONFIG_MTD_PARTITIONED_MASTER is not set
1373,1375d1547
< CONFIG_MTD_SC520CDP=m
< CONFIG_MTD_NETSC520=m
< CONFIG_MTD_TS5500=m
1406,1408d1577
< # CONFIG_MTD_DOC2000 is not set
< # CONFIG_MTD_DOC2001 is not set
< # CONFIG_MTD_DOC2001PLUS is not set
1413d1581
< # CONFIG_MTD_NAND_VERIFY_WRITE is not set
1416,1417c1584,1585
< # CONFIG_MTD_NAND_MUSEUM_IDS is not set
< # CONFIG_MTD_NAND_DENALI is not set
---
> # CONFIG_MTD_NAND_DENALI_PCI is not set
> # CONFIG_MTD_NAND_OMAP_BCH_BUILD is not set
1428c1596
< CONFIG_MTD_ALAUDA=m
---
> # CONFIG_MTD_NAND_HISI504 is not set
1432c1600
< # LPDDR flash memory drivers
---
> # LPDDR & LPDDR2 PCM memory drivers
1435a1604
> # CONFIG_MTD_SPI_NOR is not set
1438c1607,1608
< CONFIG_MTD_UBI_BEB_RESERVE=1
---
> CONFIG_MTD_UBI_BEB_LIMIT=20
> # CONFIG_MTD_UBI_FASTMAP is not set
1440c1610,1612
< # CONFIG_MTD_UBI_DEBUG is not set
---
> # CONFIG_MTD_UBI_BLOCK is not set
> # CONFIG_OF is not set
> CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
1458a1631
> # CONFIG_BLK_DEV_NULL_BLK is not set
1462d1634
< CONFIG_BLK_CPQ_DA=m
1474c1646
< # CONFIG_BLK_DEV_NVME is not set
---
> # CONFIG_BLK_DEV_SKD is not set
1477d1648
< # CONFIG_BLK_DEV_UB is not set
1481d1651
< # CONFIG_BLK_DEV_XIP is not set
1486,1487d1655
< CONFIG_XEN_BLKDEV_FRONTEND=m
< CONFIG_XEN_BLKDEV_BACKEND=m
1490a1659,1660
> # CONFIG_BLK_DEV_RSXX is not set
> # CONFIG_BLK_DEV_NVME is not set
1496a1667
> # CONFIG_DUMMY_IRQ is not set
1499d1669
< # CONFIG_INTEL_MID_PTI is not set
1515,1517c1685
< CONFIG_VMWARE_BALLOON=m
< # CONFIG_BMP085 is not set
< CONFIG_PCH_PHUB=m
---
> # CONFIG_BMP085_I2C is not set
1518a1687
> # CONFIG_SRAM is not set
1531,1533d1699
< CONFIG_IWMC3200TOP=m
< # CONFIG_IWMC3200TOP_DEBUG is not set
< CONFIG_IWMC3200TOP_DEBUGFS=y
1543a1710,1744
> # CONFIG_INTEL_MEI is not set
> # CONFIG_INTEL_MEI_ME is not set
> # CONFIG_INTEL_MEI_TXE is not set
> # CONFIG_VMWARE_VMCI is not set
> 
> #
> # Intel MIC Bus Driver
> #
> # CONFIG_INTEL_MIC_BUS is not set
> 
> #
> # SCIF Bus Driver
> #
> # CONFIG_SCIF_BUS is not set
> 
> #
> # Intel MIC Host Driver
> #
> 
> #
> # Intel MIC Card Driver
> #
> 
> #
> # SCIF Driver
> #
> 
> #
> # Intel MIC Coprocessor State Management (COSM) Drivers
> #
> # CONFIG_GENWQE is not set
> # CONFIG_ECHO is not set
> # CONFIG_CXL_BASE is not set
> # CONFIG_CXL_KERNEL_API is not set
> # CONFIG_CXL_EEH is not set
1554d1754
< CONFIG_SCSI_TGT=m
1555a1756
> # CONFIG_SCSI_MQ_DEFAULT is not set
1569d1769
< CONFIG_SCSI_MULTI_LUN=y
1573d1772
< CONFIG_SCSI_WAIT_SCAN=m
1580d1778
< CONFIG_SCSI_FC_TGT_ATTRS=y
1587d1784
< CONFIG_SCSI_SRP_TGT_ATTRS=y
1608d1804
< CONFIG_SCSI_AIC7XXX_OLD=m
1623a1820
> # CONFIG_SCSI_ESAS2R is not set
1629c1826
< CONFIG_SCSI_MPT2SAS=m
---
> CONFIG_SCSI_MPT3SAS=m
1631c1828,1829
< CONFIG_SCSI_MPT2SAS_LOGGING=y
---
> CONFIG_SCSI_MPT3SAS_MAX_SGE=128
> CONFIG_SCSI_MPT2SAS=m
1632a1831,1832
> # CONFIG_SCSI_UFSHCD_PCI is not set
> # CONFIG_SCSI_UFSHCD_PLATFORM is not set
1634a1835
> # CONFIG_SCSI_FLASHPOINT is not set
1636d1836
< CONFIG_HYPERV_STORAGE=m
1640a1841
> # CONFIG_SCSI_SNIC is not set
1659a1861
> # CONFIG_TCM_QLA2XXX is not set
1664c1866,1867
< CONFIG_SCSI_DC390T=m
---
> # CONFIG_SCSI_AM53C974 is not set
> # CONFIG_SCSI_WD719X is not set
1668d1870
< CONFIG_SCSI_SRP=m
1670a1873
> # CONFIG_SCSI_CHELSIO_FCOE is not set
1684a1888
> # CONFIG_SATA_ZPODD is not set
1724d1927
< CONFIG_PATA_ARASAN_CF=m
1729,1731d1931
< CONFIG_PATA_CS5520=m
< CONFIG_PATA_CS5530=m
< CONFIG_PATA_CS5536=m
1752d1951
< # CONFIG_PATA_SC1200 is not set
1786d1984
< # CONFIG_MULTICORE_RAID456 is not set
1788a1987,1989
> # CONFIG_MD_CLUSTER is not set
> # CONFIG_BCACHE is not set
> CONFIG_BLK_DEV_DM_BUILTIN=y
1789a1991
> # CONFIG_DM_MQ_DEFAULT is not set
1791c1993,1994
< CONFIG_DM_BUFIO=m
---
> CONFIG_DM_BUFIO=y
> CONFIG_DM_BIO_PRISON=m
1792a1996
> # CONFIG_DM_DEBUG_BLOCK_STACK_TRACING is not set
1796,1797c2000,2001
< # CONFIG_DM_DEBUG_BLOCK_STACK_TRACING is not set
< # CONFIG_DM_DEBUG_SPACE_MAPS is not set
---
> # CONFIG_DM_CACHE is not set
> # CONFIG_DM_ERA is not set
1799d2002
< CONFIG_DM_RAID=m
1800a2004
> CONFIG_DM_RAID=m
1808a2013,2014
> # CONFIG_DM_SWITCH is not set
> # CONFIG_DM_LOG_WRITES is not set
1812a2019
> # CONFIG_TCM_USER2 is not set
1815a2023
> # CONFIG_SBP_TARGET is not set
1833d2040
< # CONFIG_I2O is not set
1836a2044
> CONFIG_MII=m
1842,1844d2049
< CONFIG_MII=m
< CONFIG_IEEE802154_DRIVERS=m
< CONFIG_IEEE802154_FAKEHARD=m
1846a2052
> # CONFIG_NET_TEAM_MODE_BROADCAST is not set
1847a2054
> # CONFIG_NET_TEAM_MODE_RANDOM is not set
1848a2056
> # CONFIG_NET_TEAM_MODE_LOADBALANCE is not set
1850a2059,2061
> # CONFIG_IPVLAN is not set
> # CONFIG_VXLAN is not set
> # CONFIG_GENEVE is not set
1854d2064
< CONFIG_NETPOLL_TRAP=y
1856a2067
> # CONFIG_TUN_VNET_CROSS_LE is not set
1858a2070
> # CONFIG_NLMON is not set
1884a2097,2101
> CONFIG_VHOST_NET=m
> # CONFIG_VHOST_SCSI is not set
> CONFIG_VHOST_RING=m
> CONFIG_VHOST=m
> # CONFIG_VHOST_CROSS_ENDIAN_LEGACY is not set
1889,1893c2106,2107
< CONFIG_NET_DSA_MV88E6XXX=y
< CONFIG_NET_DSA_MV88E6060=y
< CONFIG_NET_DSA_MV88E6XXX_NEED_PPU=y
< CONFIG_NET_DSA_MV88E6131=y
< CONFIG_NET_DSA_MV88E6123_61_65=y
---
> # CONFIG_NET_DSA_MV88E6XXX is not set
> # CONFIG_NET_DSA_MV88E6XXX_NEED_PPU is not set
1902a2117,2118
> CONFIG_NET_VENDOR_AGERE=y
> # CONFIG_ET131X is not set
1905a2122
> # CONFIG_ALTERA_TSE is not set
1909a2127
> CONFIG_NET_VENDOR_ARC=y
1914a2133,2136
> # CONFIG_ALX is not set
> # CONFIG_NET_VENDOR_AURORA is not set
> CONFIG_NET_CADENCE=y
> # CONFIG_MACB is not set
1919a2142
> # CONFIG_BCMGENET is not set
1923a2147,2148
> CONFIG_BNX2X_SRIOV=y
> # CONFIG_BNXT is not set
1926c2151,2155
< CONFIG_NET_CALXEDA_XGMAC=m
---
> CONFIG_NET_VENDOR_CAVIUM=y
> # CONFIG_THUNDER_NIC_PF is not set
> # CONFIG_THUNDER_NIC_VF is not set
> # CONFIG_THUNDER_NIC_BGX is not set
> # CONFIG_LIQUIDIO is not set
1931a2161
> # CONFIG_CHELSIO_T4_DCB is not set
1934a2165
> # CONFIG_CX_ECAT is not set
1950,1951d2180
< CONFIG_DE600=m
< CONFIG_DE620=m
1956a2186,2187
> CONFIG_BE2NET_HWMON=y
> CONFIG_NET_VENDOR_EZCHIP=y
1967a2199
> CONFIG_IGB_HWMON=y
1971a2204
> CONFIG_IXGBE_HWMON=y
1974a2208,2210
> # CONFIG_I40E is not set
> # CONFIG_I40EVF is not set
> # CONFIG_FM10K is not set
1976d2211
< CONFIG_IP1000=m
1978a2214
> # CONFIG_MVMDIO is not set
1985a2222
> CONFIG_MLX4_EN_DCB=y
1987a2225,2226
> # CONFIG_MLX5_CORE is not set
> # CONFIG_MLXSW_CORE is not set
2006,2007d2244
< CONFIG_PCH_GBE=m
< # CONFIG_PCH_PTP is not set
2014a2252,2254
> CONFIG_QLCNIC_SRIOV=y
> CONFIG_QLCNIC_DCB=y
> CONFIG_QLCNIC_HWMON=y
2016a2257,2258
> # CONFIG_QED is not set
> CONFIG_NET_VENDOR_QUALCOMM=y
2025a2268
> CONFIG_NET_VENDOR_RENESAS=y
2027a2271,2273
> CONFIG_NET_VENDOR_ROCKER=y
> CONFIG_NET_VENDOR_SAMSUNG=y
> # CONFIG_SXGBE_ETH is not set
2037a2284
> CONFIG_SFC_MCDI_LOGGING=y
2040a2288
> # CONFIG_SMSC911X is not set
2046,2049d2293
< # CONFIG_STMMAC_DEBUG_FS is not set
< # CONFIG_STMMAC_DA is not set
< CONFIG_STMMAC_RING=y
< # CONFIG_STMMAC_CHAINED is not set
2054a2299
> CONFIG_NET_VENDOR_SYNOPSYS=y
2057a2303
> # CONFIG_TI_CPSW_ALE is not set
2062a2309,2311
> CONFIG_NET_VENDOR_WIZNET=y
> # CONFIG_WIZNET_W5100 is not set
> # CONFIG_WIZNET_W5300 is not set
2072a2322,2323
> # CONFIG_AQUANTIA_PHY is not set
> # CONFIG_AT803X_PHY is not set
2079a2331
> # CONFIG_TERANETICS_PHY is not set
2080a2333
> CONFIG_BCM_NET_PHYLIB=m
2081a2335,2336
> # CONFIG_BCM7XXX_PHY is not set
> # CONFIG_BCM87XX_PHY is not set
2087a2343,2345
> # CONFIG_DP83848_PHY is not set
> # CONFIG_DP83867_PHY is not set
> # CONFIG_MICROCHIP_PHY is not set
2089a2348,2349
> # CONFIG_MDIO_OCTEON is not set
> # CONFIG_MDIO_BCM_UNIMAC is not set
2108,2112c2368
< # CONFIG_TR is not set
< 
< #
< # USB Network Adapters
< #
---
> CONFIG_USB_NET_DRIVERS=y
2116a2373,2374
> # CONFIG_USB_RTL8152 is not set
> # CONFIG_USB_LAN78XX is not set
2118a2377
> CONFIG_USB_NET_AX88179_178A=m
2121a2381,2382
> # CONFIG_USB_NET_HUAWEI_CDC_NCM is not set
> # CONFIG_USB_NET_CDC_MBIM is not set
2122a2384,2385
> # CONFIG_USB_NET_SR9700 is not set
> # CONFIG_USB_NET_SR9800 is not set
2145a2409
> # CONFIG_USB_NET_CH9200 is not set
2160a2425
> CONFIG_ADM8211=m
2164d2428
< CONFIG_ADM8211=m
2167,2192c2431
< CONFIG_ATH_COMMON=m
< # CONFIG_ATH_DEBUG is not set
< CONFIG_ATH5K=m
< CONFIG_ATH5K_DEBUG=y
< # CONFIG_ATH5K_TRACER is not set
< CONFIG_ATH5K_PCI=y
< CONFIG_ATH9K_HW=m
< CONFIG_ATH9K_COMMON=m
< CONFIG_ATH9K_BTCOEX_SUPPORT=y
< CONFIG_ATH9K=m
< CONFIG_ATH9K_PCI=y
< CONFIG_ATH9K_AHB=y
< CONFIG_ATH9K_DEBUGFS=y
< # CONFIG_ATH9K_MAC_DEBUG is not set
< CONFIG_ATH9K_RATE_CONTROL=y
< CONFIG_ATH9K_HTC=m
< # CONFIG_ATH9K_HTC_DEBUGFS is not set
< CONFIG_CARL9170=m
< CONFIG_CARL9170_LEDS=y
< # CONFIG_CARL9170_DEBUGFS is not set
< CONFIG_CARL9170_WPC=y
< # CONFIG_CARL9170_HWRNG is not set
< CONFIG_ATH6KL=m
< CONFIG_ATH6KL_SDIO=m
< CONFIG_ATH6KL_USB=m
< CONFIG_ATH6KL_DEBUG=y
---
> # CONFIG_ATH_CARDS is not set
2195d2433
< # CONFIG_B43_BCMA_EXTRA is not set
2196a2435,2437
> CONFIG_B43_BUSES_BCMA_AND_SSB=y
> # CONFIG_B43_BUSES_BCMA is not set
> # CONFIG_B43_BUSES_SSB is not set
2199d2439
< CONFIG_B43_PCMCIA=y
2202a2443
> CONFIG_B43_PHY_G=y
2222a2464
> CONFIG_BRCMFMAC_PROTO_BCDC=y
2224a2467,2468
> # CONFIG_BRCMFMAC_PCIE is not set
> # CONFIG_BRCM_TRACING is not set
2243a2488,2491
> CONFIG_IWLWIFI_LEDS=y
> CONFIG_IWLDVM=m
> # CONFIG_IWLMVM is not set
> CONFIG_IWLWIFI_OPMODE_MODULAR=y
2252,2253d2499
< # CONFIG_IWLWIFI_P2P is not set
< # CONFIG_IWLWIFI_EXPERIMENTAL_MFP is not set
2263,2265d2508
< CONFIG_IWM=m
< # CONFIG_IWM_DEBUG is not set
< # CONFIG_IWM_TRACING is not set
2292a2536
> CONFIG_RT2800PCI_RT3290=y
2297a2542
> # CONFIG_RT2800USB_RT3573 is not set
2298a2544
> # CONFIG_RT2800USB_RT55XX is not set
2300a2547,2548
> CONFIG_RT2800_LIB_MMIO=m
> CONFIG_RT2X00_LIB_MMIO=m
2308a2557,2558
> # CONFIG_WL_MEDIATEK is not set
> CONFIG_RTL_CARDS=m
2311a2562,2566
> # CONFIG_RTL8723AE is not set
> # CONFIG_RTL8723BE is not set
> # CONFIG_RTL8188EE is not set
> # CONFIG_RTL8192EE is not set
> # CONFIG_RTL8821AE is not set
2313a2569,2570
> CONFIG_RTLWIFI_PCI=m
> CONFIG_RTLWIFI_USB=m
2316,2321c2573,2574
< CONFIG_WL1251=m
< CONFIG_WL1251_SDIO=m
< CONFIG_WL12XX_MENU=m
< CONFIG_WL12XX=m
< CONFIG_WL12XX_SDIO=m
< CONFIG_WL12XX_PLATFORM_DATA=y
---
> # CONFIG_RTL8XXXU is not set
> # CONFIG_WL_TI is not set
2326a2580,2582
> # CONFIG_MWIFIEX_USB is not set
> # CONFIG_CW1200 is not set
> # CONFIG_RSI_91X is not set
2333,2334d2588
< CONFIG_WIMAX_I2400M_SDIO=m
< CONFIG_WIMAX_IWMC3200_SDIO=y
2337,2338c2591
< CONFIG_XEN_NETDEV_FRONTEND=m
< CONFIG_XEN_NETDEV_BACKEND=m
---
> CONFIG_IEEE802154_DRIVERS=m
2340c2593
< CONFIG_HYPERV_NET=m
---
> # CONFIG_FUJITSU_ES is not set
2420d2672
< CONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=y
2422d2673
< CONFIG_ISDN_CAPI_MIDDLEWARE=y
2423a2675
> CONFIG_ISDN_CAPI_MIDDLEWARE=y
2424a2677
> # CONFIG_ISDN_CAPI_CAPIDRV_VERBOSE is not set
2470a2724
> # CONFIG_NVM is not set
2475a2730
> CONFIG_INPUT_LEDS=y
2478a2734
> # CONFIG_INPUT_MATRIXKMAP is not set
2503a2760
> # CONFIG_KEYBOARD_LM8333 is not set
2511d2767
< # CONFIG_KEYBOARD_OMAP4 is not set
2517a2774
> CONFIG_MOUSE_PS2_CYPRESS=y
2522a2780
> CONFIG_MOUSE_PS2_FOCALTECH=y
2525a2784,2785
> # CONFIG_MOUSE_CYAPA is not set
> # CONFIG_MOUSE_ELAN_I2C is not set
2566c2826
< CONFIG_TABLET_USB_WACOM=m
---
> # CONFIG_TABLET_SERIAL_WACOM4 is not set
2567a2828
> CONFIG_TOUCHSCREEN_PROPERTIES=y
2571a2833
> # CONFIG_TOUCHSCREEN_CYTTSP4_CORE is not set
2575d2836
< CONFIG_TOUCHSCREEN_EGALAX=m
2576a2838
> # CONFIG_TOUCHSCREEN_GOODIX is not set
2578a2841
> # CONFIG_TOUCHSCREEN_ELAN is not set
2580a2844
> # CONFIG_TOUCHSCREEN_WACOM_I2C is not set
2582a2847
> # CONFIG_TOUCHSCREEN_MMS114 is not set
2586a2852
> # CONFIG_TOUCHSCREEN_EDT_FT5X06 is not set
2589a2856
> # CONFIG_TOUCHSCREEN_WDT87XX_I2C is not set
2611a2879
> # CONFIG_TOUCHSCREEN_TSC2004 is not set
2613a2882
> # CONFIG_TOUCHSCREEN_SX8654 is not set
2614a2884
> # CONFIG_TOUCHSCREEN_ROHM_BU21023 is not set
2617a2888
> # CONFIG_INPUT_E3X0_BUTTON is not set
2632a2904
> # CONFIG_INPUT_IMS_PCU is not set
2635c2907,2909
< CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
---
> # CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set
> # CONFIG_INPUT_DRV2665_HAPTICS is not set
> # CONFIG_INPUT_DRV2667_HAPTICS is not set
2640a2915
> CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
2649a2925,2926
> # CONFIG_SERIO_ARC_PS2 is not set
> # CONFIG_USERIO is not set
2659c2936
< CONFIG_CRASH=m
---
> CONFIG_TTY=y
2682a2960
> CONFIG_DEVMEM=y
2684d2961
< # CONFIG_STALDRV is not set
2688a2966
> CONFIG_SERIAL_EARLYCON=y
2689a2968,2969
> CONFIG_SERIAL_8250_DEPRECATED_OPTIONS=y
> CONFIG_SERIAL_8250_PNP=y
2691c2971
< CONFIG_FIX_EARLYCON_MEM=y
---
> CONFIG_SERIAL_8250_DMA=y
2693d2972
< CONFIG_SERIAL_8250_PNP=y
2701a2981,2985
> # CONFIG_SERIAL_8250_FSL is not set
> # CONFIG_SERIAL_8250_DW is not set
> # CONFIG_SERIAL_8250_RT288X is not set
> # CONFIG_SERIAL_8250_FINTEK is not set
> # CONFIG_SERIAL_8250_MID is not set
2706c2990,2991
< # CONFIG_SERIAL_MFD_HSU is not set
---
> # CONFIG_SERIAL_KGDB_NMI is not set
> # CONFIG_SERIAL_UARTLITE is not set
2711c2996,2997
< # CONFIG_SERIAL_TIMBERDALE is not set
---
> # CONFIG_SERIAL_SCCNXP is not set
> # CONFIG_SERIAL_SC16IS7XX is not set
2714,2715c3000,3002
< # CONFIG_SERIAL_PCH_UART is not set
< # CONFIG_SERIAL_XILINX_PS_UART is not set
---
> # CONFIG_SERIAL_ARC is not set
> # CONFIG_SERIAL_RP2 is not set
> # CONFIG_SERIAL_FSL_LPUART is not set
2720,2722d3006
< CONFIG_HVC_IRQ=y
< CONFIG_HVC_XEN=y
< CONFIG_HVC_XEN_FRONTEND=y
2727a3012,3013
> # CONFIG_IPMI_SI_PROBE_DEFAULTS is not set
> # CONFIG_IPMI_SSIF is not set
2735a3022
> CONFIG_HW_RANDOM_TPM=m
2754a3042,3044
> # CONFIG_TCG_TIS_I2C_ATMEL is not set
> # CONFIG_TCG_TIS_I2C_INFINEON is not set
> # CONFIG_TCG_TIS_I2C_NUVOTON is not set
2757a3048
> # CONFIG_TCG_CRB is not set
2760c3051,3055
< # CONFIG_RAMOOPS is not set
---
> # CONFIG_XILLYBUS is not set
> 
> #
> # I2C support
> #
2785a3081
> # CONFIG_I2C_ISMT is not set
2804,2805d3099
< # CONFIG_I2C_EG20T is not set
< # CONFIG_I2C_INTEL_MID is not set
2817a3112
> # CONFIG_I2C_ROBOTFUZZ_OSIF is not set
2824a3120
> # CONFIG_I2C_SLAVE is not set
2828a3125
> # CONFIG_SPMI is not set
2857d3153
< CONFIG_PTP_1588_CLOCK_PCH=m
2869c3165
< # CONFIG_BATTERY_BQ27x00 is not set
---
> # CONFIG_BATTERY_BQ27XXX is not set
2874a3171
> # CONFIG_CHARGER_BQ2415X is not set
2875a3173,3175
> # CONFIG_BATTERY_GAUGE_LTC2941 is not set
> # CONFIG_POWER_RESET is not set
> # CONFIG_POWER_AVS is not set
2892a3193
> # CONFIG_SENSORS_ADT7410 is not set
2900a3202
> CONFIG_SENSORS_APPLESMC=m
2904a3207
> CONFIG_SENSORS_DELL_SMM=m
2910d3212
< CONFIG_SENSORS_G760A=m
2913c3215,3217
< CONFIG_SENSORS_CORETEMP=m
---
> CONFIG_SENSORS_G760A=m
> # CONFIG_SENSORS_G762 is not set
> # CONFIG_SENSORS_HIH6130 is not set
2915a3220,3221
> # CONFIG_SENSORS_I5500 is not set
> CONFIG_SENSORS_CORETEMP=m
2917a3224
> # CONFIG_SENSORS_POWR1220 is not set
2918a3226,3243
> # CONFIG_SENSORS_LTC2945 is not set
> CONFIG_SENSORS_LTC4151=m
> CONFIG_SENSORS_LTC4215=m
> # CONFIG_SENSORS_LTC4222 is not set
> CONFIG_SENSORS_LTC4245=m
> # CONFIG_SENSORS_LTC4260 is not set
> CONFIG_SENSORS_LTC4261=m
> CONFIG_SENSORS_MAX16065=m
> CONFIG_SENSORS_MAX1619=m
> CONFIG_SENSORS_MAX1668=m
> # CONFIG_SENSORS_MAX197 is not set
> CONFIG_SENSORS_MAX6639=m
> CONFIG_SENSORS_MAX6642=m
> CONFIG_SENSORS_MAX6650=m
> # CONFIG_SENSORS_MAX6697 is not set
> # CONFIG_SENSORS_MAX31790 is not set
> # CONFIG_SENSORS_HTU21 is not set
> CONFIG_SENSORS_MCP3021=m
2931,2934c3256
< CONFIG_SENSORS_LTC4151=m
< CONFIG_SENSORS_LTC4215=m
< CONFIG_SENSORS_LTC4245=m
< CONFIG_SENSORS_LTC4261=m
---
> # CONFIG_SENSORS_LM95234 is not set
2937,2944d3258
< CONFIG_SENSORS_MAX16065=m
< CONFIG_SENSORS_MAX1619=m
< CONFIG_SENSORS_MAX1668=m
< CONFIG_SENSORS_MAX6639=m
< CONFIG_SENSORS_MAX6642=m
< CONFIG_SENSORS_MAX6650=m
< CONFIG_SENSORS_MCP3021=m
< CONFIG_SENSORS_NTC_THERMISTOR=m
2946a3261,3265
> CONFIG_SENSORS_NTC_THERMISTOR=m
> # CONFIG_SENSORS_NCT6683 is not set
> # CONFIG_SENSORS_NCT6775 is not set
> # CONFIG_SENSORS_NCT7802 is not set
> # CONFIG_SENSORS_NCT7904 is not set
2953a3273
> # CONFIG_SENSORS_MAX20751 is not set
2955a3276
> # CONFIG_SENSORS_TPS40422 is not set
2959a3281
> # CONFIG_SENSORS_SHTC1 is not set
2961d3282
< # CONFIG_SENSORS_SMM665 is not set
2971a3293,3294
> # CONFIG_SENSORS_SMM665 is not set
> # CONFIG_SENSORS_ADC128D818 is not set
2974a3298,3300
> # CONFIG_SENSORS_INA209 is not set
> # CONFIG_SENSORS_INA2XX is not set
> # CONFIG_SENSORS_TC74 is not set
2976a3303
> # CONFIG_SENSORS_TMP103 is not set
2993d3319
< CONFIG_SENSORS_APPLESMC=m
3001a3328,3343
> CONFIG_THERMAL_WRITABLE_TRIPS=y
> CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
> # CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set
> # CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE is not set
> # CONFIG_THERMAL_DEFAULT_GOV_POWER_ALLOCATOR is not set
> # CONFIG_THERMAL_GOV_FAIR_SHARE is not set
> CONFIG_THERMAL_GOV_STEP_WISE=y
> CONFIG_THERMAL_GOV_BANG_BANG=y
> CONFIG_THERMAL_GOV_USER_SPACE=y
> # CONFIG_THERMAL_GOV_POWER_ALLOCATOR is not set
> # CONFIG_THERMAL_EMULATION is not set
> # CONFIG_INTEL_POWERCLAMP is not set
> CONFIG_X86_PKG_TEMP_THERMAL=m
> # CONFIG_INTEL_SOC_DTS_THERMAL is not set
> # CONFIG_INT340X_THERMAL is not set
> # CONFIG_INTEL_PCH_THERMAL is not set
3009a3352,3355
> # CONFIG_XILINX_WATCHDOG is not set
> # CONFIG_CADENCE_WATCHDOG is not set
> # CONFIG_DW_WATCHDOG is not set
> # CONFIG_MAX63XX_WATCHDOG is not set
3016d3361
< # CONFIG_SC520_WDT is not set
3022a3368
> # CONFIG_IE6XX_WDT is not set
3033d3378
< # CONFIG_SBC8360_WDT is not set
3039,3040d3383
< CONFIG_W83697HF_WDT=m
< CONFIG_W83697UG_WDT=m
3045c3388
< CONFIG_XEN_WDT=m
---
> # CONFIG_BCM7038_WDT is not set
3071a3415
> # CONFIG_SSB_HOST_SOC is not set
3083a3428,3430
> # CONFIG_BCMA_HOST_SOC is not set
> CONFIG_BCMA_DRIVER_PCI=y
> # CONFIG_BCMA_DRIVER_GMAC_CMN is not set
3090c3437,3440
< CONFIG_MFD_SM501=m
---
> # CONFIG_MFD_BCM590XX is not set
> # CONFIG_MFD_CROS_EC is not set
> # CONFIG_MFD_DLN2 is not set
> # CONFIG_MFD_MC13XXX_I2C is not set
3091a3442,3463
> CONFIG_LPC_ICH=m
> CONFIG_LPC_SCH=m
> # CONFIG_MFD_INTEL_LPSS_ACPI is not set
> # CONFIG_MFD_INTEL_LPSS_PCI is not set
> # CONFIG_MFD_JANZ_CMODIO is not set
> # CONFIG_MFD_KEMPLD is not set
> # CONFIG_MFD_MT6397 is not set
> # CONFIG_MFD_MENF21BMC is not set
> # CONFIG_MFD_VIPERBOARD is not set
> # CONFIG_MFD_RETU is not set
> # CONFIG_MFD_PCF50633 is not set
> # CONFIG_MFD_RDC321X is not set
> # CONFIG_MFD_RTSX_PCI is not set
> # CONFIG_MFD_RTSX_USB is not set
> # CONFIG_MFD_RN5T618 is not set
> # CONFIG_MFD_SI476X_CORE is not set
> CONFIG_MFD_SM501=m
> # CONFIG_MFD_SKY81452 is not set
> # CONFIG_ABX500_CORE is not set
> # CONFIG_MFD_SYSCON is not set
> # CONFIG_MFD_TI_AM335X_TSCADC is not set
> # CONFIG_MFD_LP3943 is not set
3094a3467,3469
> # CONFIG_MFD_TPS65218 is not set
> CONFIG_MFD_WL1273_CORE=m
> # CONFIG_MFD_LM3533 is not set
3096,3102d3470
< CONFIG_MFD_WM8400=m
< # CONFIG_MFD_PCF50633 is not set
< # CONFIG_ABX500_CORE is not set
< # CONFIG_MFD_CS5535 is not set
< CONFIG_LPC_SCH=m
< # CONFIG_MFD_RDC321X is not set
< # CONFIG_MFD_JANZ_CMODIO is not set
3104c3472,3473
< CONFIG_MFD_WL1273_CORE=m
---
> # CONFIG_MFD_ARIZONA_I2C is not set
> # CONFIG_MFD_WM8994 is not set
3111,3184c3480,3485
< CONFIG_MEDIA_CONTROLLER=y
< CONFIG_VIDEO_DEV=m
< CONFIG_VIDEO_V4L2_COMMON=m
< CONFIG_VIDEO_V4L2_SUBDEV_API=y
< CONFIG_DVB_CORE=m
< CONFIG_DVB_NET=y
< CONFIG_VIDEO_MEDIA=m
< 
< #
< # Multimedia drivers
< #
< CONFIG_VIDEO_SAA7146=m
< CONFIG_VIDEO_SAA7146_VV=m
< CONFIG_RC_CORE=m
< CONFIG_LIRC=m
< CONFIG_RC_MAP=m
< CONFIG_IR_NEC_DECODER=m
< CONFIG_IR_RC5_DECODER=m
< CONFIG_IR_RC6_DECODER=m
< CONFIG_IR_JVC_DECODER=m
< CONFIG_IR_SONY_DECODER=m
< CONFIG_IR_RC5_SZ_DECODER=m
< CONFIG_IR_SANYO_DECODER=m
< CONFIG_IR_MCE_KBD_DECODER=m
< CONFIG_IR_LIRC_CODEC=m
< CONFIG_RC_ATI_REMOTE=m
< CONFIG_IR_ENE=m
< CONFIG_IR_IMON=m
< CONFIG_IR_MCEUSB=m
< CONFIG_IR_ITE_CIR=m
< CONFIG_IR_FINTEK=m
< CONFIG_IR_NUVOTON=m
< CONFIG_IR_REDRAT3=m
< CONFIG_IR_STREAMZAP=m
< CONFIG_IR_WINBOND_CIR=m
< CONFIG_RC_LOOPBACK=m
< CONFIG_IR_GPIO_CIR=m
< CONFIG_MEDIA_ATTACH=y
< CONFIG_MEDIA_TUNER=m
< # CONFIG_MEDIA_TUNER_CUSTOMISE is not set
< CONFIG_MEDIA_TUNER_SIMPLE=m
< CONFIG_MEDIA_TUNER_TDA8290=m
< CONFIG_MEDIA_TUNER_TDA827X=m
< CONFIG_MEDIA_TUNER_TDA18271=m
< CONFIG_MEDIA_TUNER_TDA9887=m
< CONFIG_MEDIA_TUNER_TEA5761=m
< CONFIG_MEDIA_TUNER_TEA5767=m
< CONFIG_MEDIA_TUNER_MT20XX=m
< CONFIG_MEDIA_TUNER_MT2060=m
< CONFIG_MEDIA_TUNER_MT2063=m
< CONFIG_MEDIA_TUNER_MT2266=m
< CONFIG_MEDIA_TUNER_MT2131=m
< CONFIG_MEDIA_TUNER_QT1010=m
< CONFIG_MEDIA_TUNER_XC2028=m
< CONFIG_MEDIA_TUNER_XC5000=m
< CONFIG_MEDIA_TUNER_XC4000=m
< CONFIG_MEDIA_TUNER_MXL5005S=m
< CONFIG_MEDIA_TUNER_MXL5007T=m
< CONFIG_MEDIA_TUNER_MC44S803=m
< CONFIG_MEDIA_TUNER_MAX2165=m
< CONFIG_MEDIA_TUNER_TDA18218=m
< CONFIG_MEDIA_TUNER_TDA18212=m
< CONFIG_VIDEO_V4L2=m
< CONFIG_VIDEOBUF_GEN=m
< CONFIG_VIDEOBUF_DMA_SG=m
< CONFIG_VIDEOBUF_VMALLOC=m
< CONFIG_VIDEOBUF_DVB=m
< CONFIG_VIDEO_BTCX=m
< CONFIG_VIDEO_TVEEPROM=m
< CONFIG_VIDEO_TUNER=m
< CONFIG_VIDEOBUF2_CORE=m
< CONFIG_VIDEOBUF2_MEMOPS=m
< CONFIG_VIDEOBUF2_VMALLOC=m
< CONFIG_VIDEO_CAPTURE_DRIVERS=y
---
> # CONFIG_MEDIA_CAMERA_SUPPORT is not set
> # CONFIG_MEDIA_ANALOG_TV_SUPPORT is not set
> # CONFIG_MEDIA_DIGITAL_TV_SUPPORT is not set
> # CONFIG_MEDIA_RADIO_SUPPORT is not set
> # CONFIG_MEDIA_SDR_SUPPORT is not set
> # CONFIG_MEDIA_RC_SUPPORT is not set
3187,3188c3488
< CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
< CONFIG_VIDEO_IR_I2C=m
---
> # CONFIG_TTPCI_EEPROM is not set
3191c3491
< # Audio decoders, processors and mixers
---
> # Media drivers
3193,3609c3493,3494
< CONFIG_VIDEO_TVAUDIO=m
< CONFIG_VIDEO_TDA7432=m
< CONFIG_VIDEO_TDA9840=m
< CONFIG_VIDEO_TEA6415C=m
< CONFIG_VIDEO_TEA6420=m
< CONFIG_VIDEO_MSP3400=m
< CONFIG_VIDEO_CS5345=m
< CONFIG_VIDEO_CS53L32A=m
< CONFIG_VIDEO_WM8775=m
< CONFIG_VIDEO_WM8739=m
< CONFIG_VIDEO_VP27SMPX=m
< 
< #
< # RDS decoders
< #
< CONFIG_VIDEO_SAA6588=m
< 
< #
< # Video decoders
< #
< CONFIG_VIDEO_BT819=m
< CONFIG_VIDEO_BT856=m
< CONFIG_VIDEO_BT866=m
< CONFIG_VIDEO_KS0127=m
< CONFIG_VIDEO_SAA7110=m
< CONFIG_VIDEO_SAA711X=m
< CONFIG_VIDEO_TVP5150=m
< CONFIG_VIDEO_VPX3220=m
< 
< #
< # Video and audio decoders
< #
< CONFIG_VIDEO_SAA717X=m
< CONFIG_VIDEO_CX25840=m
< 
< #
< # MPEG video encoders
< #
< CONFIG_VIDEO_CX2341X=m
< 
< #
< # Video encoders
< #
< CONFIG_VIDEO_SAA7127=m
< CONFIG_VIDEO_SAA7185=m
< CONFIG_VIDEO_ADV7170=m
< CONFIG_VIDEO_ADV7175=m
< 
< #
< # Camera sensor devices
< #
< CONFIG_VIDEO_MT9V011=m
< 
< #
< # Flash devices
< #
< 
< #
< # Video improvement chips
< #
< CONFIG_VIDEO_UPD64031A=m
< CONFIG_VIDEO_UPD64083=m
< 
< #
< # Miscelaneous helper chips
< #
< CONFIG_VIDEO_M52790=m
< # CONFIG_VIDEO_VIVI is not set
< CONFIG_V4L_USB_DRIVERS=y
< CONFIG_USB_VIDEO_CLASS=m
< CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
< CONFIG_USB_GSPCA=m
< CONFIG_USB_M5602=m
< CONFIG_USB_STV06XX=m
< CONFIG_USB_GL860=m
< CONFIG_USB_GSPCA_BENQ=m
< CONFIG_USB_GSPCA_CONEX=m
< CONFIG_USB_GSPCA_CPIA1=m
< CONFIG_USB_GSPCA_ETOMS=m
< CONFIG_USB_GSPCA_FINEPIX=m
< CONFIG_USB_GSPCA_JEILINJ=m
< CONFIG_USB_GSPCA_JL2005BCD=m
< CONFIG_USB_GSPCA_KINECT=m
< CONFIG_USB_GSPCA_KONICA=m
< CONFIG_USB_GSPCA_MARS=m
< CONFIG_USB_GSPCA_MR97310A=m
< CONFIG_USB_GSPCA_NW80X=m
< CONFIG_USB_GSPCA_OV519=m
< CONFIG_USB_GSPCA_OV534=m
< CONFIG_USB_GSPCA_OV534_9=m
< CONFIG_USB_GSPCA_PAC207=m
< CONFIG_USB_GSPCA_PAC7302=m
< CONFIG_USB_GSPCA_PAC7311=m
< CONFIG_USB_GSPCA_SE401=m
< CONFIG_USB_GSPCA_SN9C2028=m
< CONFIG_USB_GSPCA_SN9C20X=m
< CONFIG_USB_GSPCA_SONIXB=m
< CONFIG_USB_GSPCA_SONIXJ=m
< CONFIG_USB_GSPCA_SPCA500=m
< CONFIG_USB_GSPCA_SPCA501=m
< CONFIG_USB_GSPCA_SPCA505=m
< CONFIG_USB_GSPCA_SPCA506=m
< CONFIG_USB_GSPCA_SPCA508=m
< CONFIG_USB_GSPCA_SPCA561=m
< CONFIG_USB_GSPCA_SPCA1528=m
< CONFIG_USB_GSPCA_SQ905=m
< CONFIG_USB_GSPCA_SQ905C=m
< CONFIG_USB_GSPCA_SQ930X=m
< CONFIG_USB_GSPCA_STK014=m
< CONFIG_USB_GSPCA_STV0680=m
< CONFIG_USB_GSPCA_SUNPLUS=m
< CONFIG_USB_GSPCA_T613=m
< CONFIG_USB_GSPCA_TOPRO=m
< CONFIG_USB_GSPCA_TV8532=m
< CONFIG_USB_GSPCA_VC032X=m
< CONFIG_USB_GSPCA_VICAM=m
< CONFIG_USB_GSPCA_XIRLINK_CIT=m
< CONFIG_USB_GSPCA_ZC3XX=m
< CONFIG_VIDEO_PVRUSB2=m
< CONFIG_VIDEO_PVRUSB2_SYSFS=y
< CONFIG_VIDEO_PVRUSB2_DVB=y
< # CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set
< CONFIG_VIDEO_HDPVR=m
< CONFIG_VIDEO_EM28XX=m
< CONFIG_VIDEO_EM28XX_ALSA=m
< CONFIG_VIDEO_EM28XX_DVB=m
< CONFIG_VIDEO_EM28XX_RC=y
< CONFIG_VIDEO_TLG2300=m
< CONFIG_VIDEO_CX231XX=m
< CONFIG_VIDEO_CX231XX_RC=y
< CONFIG_VIDEO_CX231XX_ALSA=m
< CONFIG_VIDEO_CX231XX_DVB=m
< CONFIG_VIDEO_TM6000=m
< CONFIG_VIDEO_TM6000_ALSA=m
< CONFIG_VIDEO_TM6000_DVB=m
< CONFIG_VIDEO_USBVISION=m
< # CONFIG_USB_ET61X251 is not set
< # CONFIG_USB_SN9C102 is not set
< CONFIG_USB_PWC=m
< # CONFIG_USB_PWC_DEBUG is not set
< CONFIG_USB_PWC_INPUT_EVDEV=y
< CONFIG_VIDEO_CPIA2=m
< CONFIG_USB_ZR364XX=m
< CONFIG_USB_STKWEBCAM=m
< CONFIG_USB_S2255=m
< CONFIG_V4L_PCI_DRIVERS=y
< CONFIG_VIDEO_AU0828=m
< CONFIG_VIDEO_BT848=m
< CONFIG_VIDEO_BT848_DVB=y
< CONFIG_VIDEO_CX18=m
< CONFIG_VIDEO_CX18_ALSA=m
< CONFIG_VIDEO_CX23885=m
< CONFIG_MEDIA_ALTERA_CI=m
< # CONFIG_VIDEO_CX25821 is not set
< CONFIG_VIDEO_CX88=m
< CONFIG_VIDEO_CX88_ALSA=m
< CONFIG_VIDEO_CX88_BLACKBIRD=m
< CONFIG_VIDEO_CX88_DVB=m
< CONFIG_VIDEO_CX88_VP3054=m
< CONFIG_VIDEO_CX88_MPEG=m
< CONFIG_VIDEO_HEXIUM_GEMINI=m
< CONFIG_VIDEO_HEXIUM_ORION=m
< CONFIG_VIDEO_IVTV=m
< CONFIG_VIDEO_FB_IVTV=m
< CONFIG_VIDEO_MEYE=m
< CONFIG_VIDEO_MXB=m
< CONFIG_VIDEO_SAA7134=m
< CONFIG_VIDEO_SAA7134_ALSA=m
< CONFIG_VIDEO_SAA7134_RC=y
< CONFIG_VIDEO_SAA7134_DVB=m
< CONFIG_VIDEO_SAA7164=m
< CONFIG_VIDEO_ZORAN=m
< CONFIG_VIDEO_ZORAN_DC30=m
< CONFIG_VIDEO_ZORAN_ZR36060=m
< CONFIG_VIDEO_ZORAN_BUZ=m
< CONFIG_VIDEO_ZORAN_DC10=m
< CONFIG_VIDEO_ZORAN_LML33=m
< CONFIG_VIDEO_ZORAN_LML33R10=m
< CONFIG_VIDEO_ZORAN_AVS6EYES=m
< # CONFIG_V4L_ISA_PARPORT_DRIVERS is not set
< # CONFIG_V4L_PLATFORM_DRIVERS is not set
< CONFIG_V4L_MEM2MEM_DRIVERS=y
< # CONFIG_VIDEO_MEM2MEM_TESTDEV is not set
< CONFIG_RADIO_ADAPTERS=y
< CONFIG_RADIO_SI470X=y
< CONFIG_USB_SI470X=m
< CONFIG_I2C_SI470X=m
< CONFIG_USB_MR800=m
< CONFIG_USB_DSBR=m
< CONFIG_RADIO_MAXIRADIO=m
< CONFIG_I2C_SI4713=m
< CONFIG_RADIO_SI4713=m
< # CONFIG_USB_KEENE is not set
< CONFIG_RADIO_TEA5764=m
< CONFIG_RADIO_SAA7706H=m
< # CONFIG_RADIO_TEF6862 is not set
< CONFIG_RADIO_WL1273=m
< 
< #
< # Texas Instruments WL128x FM driver (ST based)
< #
< CONFIG_DVB_MAX_ADAPTERS=8
< CONFIG_DVB_DYNAMIC_MINORS=y
< CONFIG_DVB_CAPTURE_DRIVERS=y
< 
< #
< # Supported SAA7146 based PCI Adapters
< #
< CONFIG_TTPCI_EEPROM=m
< CONFIG_DVB_AV7110=m
< CONFIG_DVB_AV7110_OSD=y
< CONFIG_DVB_BUDGET_CORE=m
< CONFIG_DVB_BUDGET=m
< CONFIG_DVB_BUDGET_CI=m
< CONFIG_DVB_BUDGET_AV=m
< CONFIG_DVB_BUDGET_PATCH=m
< 
< #
< # Supported USB Adapters
< #
< CONFIG_DVB_USB=m
< # CONFIG_DVB_USB_DEBUG is not set
< CONFIG_DVB_USB_A800=m
< CONFIG_DVB_USB_DIBUSB_MB=m
< # CONFIG_DVB_USB_DIBUSB_MB_FAULTY is not set
< CONFIG_DVB_USB_DIBUSB_MC=m
< CONFIG_DVB_USB_DIB0700=m
< CONFIG_DVB_USB_UMT_010=m
< CONFIG_DVB_USB_CXUSB=m
< CONFIG_DVB_USB_M920X=m
< CONFIG_DVB_USB_GL861=m
< CONFIG_DVB_USB_AU6610=m
< CONFIG_DVB_USB_DIGITV=m
< CONFIG_DVB_USB_VP7045=m
< CONFIG_DVB_USB_VP702X=m
< CONFIG_DVB_USB_GP8PSK=m
< CONFIG_DVB_USB_NOVA_T_USB2=m
< CONFIG_DVB_USB_TTUSB2=m
< CONFIG_DVB_USB_DTT200U=m
< CONFIG_DVB_USB_OPERA1=m
< CONFIG_DVB_USB_AF9005=m
< CONFIG_DVB_USB_AF9005_REMOTE=m
< CONFIG_DVB_USB_PCTV452E=m
< CONFIG_DVB_USB_DW2102=m
< CONFIG_DVB_USB_CINERGY_T2=m
< CONFIG_DVB_USB_ANYSEE=m
< CONFIG_DVB_USB_DTV5100=m
< CONFIG_DVB_USB_AF9015=m
< CONFIG_DVB_USB_CE6230=m
< CONFIG_DVB_USB_FRIIO=m
< CONFIG_DVB_USB_EC168=m
< CONFIG_DVB_USB_AZ6007=m
< CONFIG_DVB_USB_AZ6027=m
< CONFIG_DVB_USB_LME2510=m
< CONFIG_DVB_USB_TECHNISAT_USB2=m
< CONFIG_DVB_USB_IT913X=m
< CONFIG_DVB_USB_MXL111SF=m
< CONFIG_DVB_USB_RTL28XXU=m
< CONFIG_DVB_TTUSB_BUDGET=m
< CONFIG_DVB_TTUSB_DEC=m
< CONFIG_SMS_SIANO_MDTV=m
< 
< #
< # Siano module components
< #
< CONFIG_SMS_USB_DRV=m
< CONFIG_SMS_SDIO_DRV=m
< 
< #
< # Supported FlexCopII (B2C2) Adapters
< #
< CONFIG_DVB_B2C2_FLEXCOP=m
< CONFIG_DVB_B2C2_FLEXCOP_PCI=m
< CONFIG_DVB_B2C2_FLEXCOP_USB=m
< # CONFIG_DVB_B2C2_FLEXCOP_DEBUG is not set
< 
< #
< # Supported BT878 Adapters
< #
< CONFIG_DVB_BT8XX=m
< 
< #
< # Supported Pluto2 Adapters
< #
< CONFIG_DVB_PLUTO2=m
< 
< #
< # Supported SDMC DM1105 Adapters
< #
< CONFIG_DVB_DM1105=m
< 
< #
< # Supported FireWire (IEEE 1394) Adapters
< #
< CONFIG_DVB_FIREDTV=m
< CONFIG_DVB_FIREDTV_INPUT=y
< 
< #
< # Supported Earthsoft PT1 Adapters
< #
< CONFIG_DVB_PT1=m
< 
< #
< # Supported Mantis Adapters
< #
< CONFIG_MANTIS_CORE=m
< CONFIG_DVB_MANTIS=m
< CONFIG_DVB_HOPPER=m
< 
< #
< # Supported nGene Adapters
< #
< CONFIG_DVB_NGENE=m
< 
< #
< # Supported ddbridge ('Octopus') Adapters
< #
< CONFIG_DVB_DDBRIDGE=m
< 
< #
< # Supported DVB Frontends
< #
< # CONFIG_DVB_FE_CUSTOMISE is not set
< 
< #
< # Multistandard (satellite) frontends
< #
< CONFIG_DVB_STB0899=m
< CONFIG_DVB_STB6100=m
< CONFIG_DVB_STV090x=m
< CONFIG_DVB_STV6110x=m
< 
< #
< # Multistandard (cable + terrestrial) frontends
< #
< CONFIG_DVB_DRXK=m
< CONFIG_DVB_TDA18271C2DD=m
< 
< #
< # DVB-S (satellite) frontends
< #
< CONFIG_DVB_CX24110=m
< CONFIG_DVB_CX24123=m
< CONFIG_DVB_MT312=m
< CONFIG_DVB_ZL10036=m
< CONFIG_DVB_ZL10039=m
< CONFIG_DVB_S5H1420=m
< CONFIG_DVB_STV0288=m
< CONFIG_DVB_STB6000=m
< CONFIG_DVB_STV0299=m
< CONFIG_DVB_STV6110=m
< CONFIG_DVB_STV0900=m
< CONFIG_DVB_TDA8083=m
< CONFIG_DVB_TDA10086=m
< CONFIG_DVB_TDA8261=m
< CONFIG_DVB_VES1X93=m
< CONFIG_DVB_TUNER_ITD1000=m
< CONFIG_DVB_TUNER_CX24113=m
< CONFIG_DVB_TDA826X=m
< CONFIG_DVB_TUA6100=m
< CONFIG_DVB_CX24116=m
< CONFIG_DVB_SI21XX=m
< CONFIG_DVB_DS3000=m
< CONFIG_DVB_MB86A16=m
< CONFIG_DVB_TDA10071=m
< 
< #
< # DVB-T (terrestrial) frontends
< #
< CONFIG_DVB_SP8870=m
< CONFIG_DVB_SP887X=m
< CONFIG_DVB_CX22700=m
< CONFIG_DVB_CX22702=m
< CONFIG_DVB_DRXD=m
< CONFIG_DVB_L64781=m
< CONFIG_DVB_TDA1004X=m
< CONFIG_DVB_NXT6000=m
< CONFIG_DVB_MT352=m
< CONFIG_DVB_ZL10353=m
< CONFIG_DVB_DIB3000MB=m
< CONFIG_DVB_DIB3000MC=m
< CONFIG_DVB_DIB7000M=m
< CONFIG_DVB_DIB7000P=m
< CONFIG_DVB_TDA10048=m
< CONFIG_DVB_AF9013=m
< CONFIG_DVB_EC100=m
< CONFIG_DVB_STV0367=m
< CONFIG_DVB_CXD2820R=m
< CONFIG_DVB_RTL2830=m
< 
< #
< # DVB-C (cable) frontends
< #
< CONFIG_DVB_VES1820=m
< CONFIG_DVB_TDA10021=m
< CONFIG_DVB_TDA10023=m
< CONFIG_DVB_STV0297=m
< 
< #
< # ATSC (North American/Korean Terrestrial/Cable DTV) frontends
< #
< CONFIG_DVB_NXT200X=m
< CONFIG_DVB_OR51211=m
< CONFIG_DVB_OR51132=m
< CONFIG_DVB_BCM3510=m
< CONFIG_DVB_LGDT330X=m
< CONFIG_DVB_LGDT3305=m
< CONFIG_DVB_S5H1409=m
< CONFIG_DVB_AU8522=m
< CONFIG_DVB_S5H1411=m
< 
< #
< # ISDB-T (terrestrial) frontends
< #
< CONFIG_DVB_S921=m
< CONFIG_DVB_DIB8000=m
< CONFIG_DVB_MB86A20S=m
---
> # CONFIG_MEDIA_USB_SUPPORT is not set
> # CONFIG_MEDIA_PCI_SUPPORT is not set
3612c3497,3502
< # Digital terrestrial only tuners/PLL
---
> # Supported MMC/SDIO adapters
> #
> # CONFIG_CYPRESS_FIRMWARE is not set
> 
> #
> # Media ancillary drivers (tuners, sensors, i2c, frontends)
3614,3616d3503
< CONFIG_DVB_PLL=m
< CONFIG_DVB_TUNER_DIB0070=m
< CONFIG_DVB_TUNER_DIB0090=m
3619c3506
< # SEC control devices for DVB-S
---
> # Customise DVB Frontends
3621,3632c3508,3509
< CONFIG_DVB_LNBP21=m
< CONFIG_DVB_LNBP22=m
< CONFIG_DVB_ISL6405=m
< CONFIG_DVB_ISL6421=m
< CONFIG_DVB_ISL6423=m
< CONFIG_DVB_A8293=m
< CONFIG_DVB_LGS8GXX=m
< CONFIG_DVB_ATBM8830=m
< CONFIG_DVB_TDA665x=m
< CONFIG_DVB_IX2505V=m
< CONFIG_DVB_IT913X_FE=m
< CONFIG_DVB_M88RS2000=m
---
> CONFIG_DVB_TUNER_DIB0070=m
> CONFIG_DVB_TUNER_DIB0090=m
3646a3524
> CONFIG_INTEL_GTT=y
3651c3529
< CONFIG_DRM_USB=m
---
> CONFIG_DRM_MIPI_DSI=y
3652a3531,3532
> CONFIG_DRM_KMS_FB_HELPER=y
> CONFIG_DRM_FBDEV_EMULATION=y
3655,3661d3534
< CONFIG_DRM_TDFX=m
< CONFIG_DRM_R128=m
< CONFIG_DRM_RADEON=m
< CONFIG_DRM_RADEON_KMS=y
< CONFIG_DRM_NOUVEAU=m
< CONFIG_DRM_NOUVEAU_BACKLIGHT=y
< CONFIG_DRM_NOUVEAU_DEBUG=y
3665a3539
> # CONFIG_DRM_I2C_ADV7511 is not set
3667a3542,3552
> # CONFIG_DRM_I2C_NXP_TDA998X is not set
> CONFIG_DRM_TDFX=m
> CONFIG_DRM_R128=m
> CONFIG_DRM_RADEON=m
> # CONFIG_DRM_RADEON_USERPTR is not set
> # CONFIG_DRM_RADEON_UMS is not set
> # CONFIG_DRM_AMDGPU is not set
> CONFIG_DRM_NOUVEAU=m
> CONFIG_NOUVEAU_DEBUG=5
> CONFIG_NOUVEAU_DEBUG_DEFAULT=3
> CONFIG_DRM_NOUVEAU_BACKLIGHT=y
3670c3555
< CONFIG_DRM_I915_KMS=y
---
> # CONFIG_DRM_I915_PRELIMINARY_HW_SUPPORT is not set
3674a3560
> # CONFIG_DRM_VGEM is not set
3675a3562
> # CONFIG_DRM_VMWGFX_FBCON is not set
3680,3682c3567,3587
< CONFIG_STUB_POULSBO=m
< CONFIG_VGASTATE=m
< CONFIG_VIDEO_OUTPUT_CONTROL=m
---
> # CONFIG_DRM_AST is not set
> # CONFIG_DRM_MGAG200 is not set
> # CONFIG_DRM_CIRRUS_QEMU is not set
> # CONFIG_DRM_QXL is not set
> # CONFIG_DRM_BOCHS is not set
> # CONFIG_DRM_VIRTIO_GPU is not set
> CONFIG_DRM_PANEL=y
> 
> #
> # Display Panels
> #
> CONFIG_DRM_BRIDGE=y
> 
> #
> # Display Interface Bridges
> #
> # CONFIG_HSA_AMD is not set
> 
> #
> # Frame buffer Devices
> #
3684a3590
> CONFIG_FB_CMDLINE=y
3691,3693c3597,3599
< CONFIG_FB_SYS_FILLRECT=y
< CONFIG_FB_SYS_COPYAREA=y
< CONFIG_FB_SYS_IMAGEBLIT=y
---
> CONFIG_FB_SYS_FILLRECT=m
> CONFIG_FB_SYS_COPYAREA=m
> CONFIG_FB_SYS_IMAGEBLIT=m
3695,3696c3601
< CONFIG_FB_SYS_FOPS=y
< # CONFIG_FB_WMT_GE_ROPS is not set
---
> CONFIG_FB_SYS_FOPS=m
3718a3624
> # CONFIG_FB_OPENCORES is not set
3731d3636
< # CONFIG_FB_VIA is not set
3741,3742d3645
< # CONFIG_FB_GEODE is not set
< # CONFIG_FB_TMIO is not set
3745a3649
> # CONFIG_FB_IBM_GXT4500 is not set
3747d3650
< CONFIG_XEN_FBDEV_FRONTEND=y
3751c3654,3656
< # CONFIG_EXYNOS_VIDEO is not set
---
> # CONFIG_FB_AUO_K190X is not set
> # CONFIG_FB_SIMPLE is not set
> # CONFIG_FB_SM712 is not set
3757d3661
< CONFIG_BACKLIGHT_PROGEAR=m
3758a3663
> # CONFIG_BACKLIGHT_PM8941_WLED is not set
3762c3667,3671
< CONFIG_BACKLIGHT_LP855X=m
---
> # CONFIG_BACKLIGHT_LM3639 is not set
> # CONFIG_BACKLIGHT_LV5207LP is not set
> # CONFIG_BACKLIGHT_BD6107 is not set
> CONFIG_VGASTATE=m
> CONFIG_HDMI=y
3770a3680,3681
> CONFIG_DUMMY_CONSOLE_COLUMNS=80
> CONFIG_DUMMY_CONSOLE_ROWS=25
3774,3776d3684
< # CONFIG_FONTS is not set
< CONFIG_FONT_8x8=y
< CONFIG_FONT_8x16=y
3795a3704
> CONFIG_SND_PCM_TIMER=y
3799a3709
> CONFIG_SND_MAX_CARDS=32
3800a3711
> CONFIG_SND_PROC_FS=y
3807d3717
< CONFIG_SND_KCTL_JACK=y
3831,3832d3740
< CONFIG_SND_SB16_DSP=m
< CONFIG_SND_TEA575X=m
3855,3856d3762
< CONFIG_SND_CS5530=m
< CONFIG_SND_CS5535AUDIO=m
3879d3784
< CONFIG_SND_ES1968_RADIO=y
3881,3904d3785
< CONFIG_SND_FM801_TEA575X_BOOL=y
< CONFIG_SND_HDA_INTEL=m
< CONFIG_SND_HDA_PREALLOC_SIZE=4096
< CONFIG_SND_HDA_HWDEP=y
< CONFIG_SND_HDA_RECONFIG=y
< CONFIG_SND_HDA_INPUT_BEEP=y
< CONFIG_SND_HDA_INPUT_BEEP_MODE=0
< CONFIG_SND_HDA_INPUT_JACK=y
< CONFIG_SND_HDA_PATCH_LOADER=y
< CONFIG_SND_HDA_CODEC_REALTEK=y
< CONFIG_SND_HDA_ENABLE_REALTEK_QUIRKS=y
< CONFIG_SND_HDA_CODEC_ANALOG=y
< CONFIG_SND_HDA_CODEC_SIGMATEL=y
< CONFIG_SND_HDA_CODEC_VIA=y
< CONFIG_SND_HDA_CODEC_HDMI=y
< CONFIG_SND_HDA_CODEC_CIRRUS=y
< CONFIG_SND_HDA_CODEC_CONEXANT=y
< CONFIG_SND_HDA_CODEC_CA0110=y
< CONFIG_SND_HDA_CODEC_CA0132=y
< CONFIG_SND_HDA_CODEC_CMEDIA=y
< CONFIG_SND_HDA_CODEC_SI3054=y
< CONFIG_SND_HDA_GENERIC=y
< CONFIG_SND_HDA_POWER_SAVE=y
< CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
3929a3811,3838
> 
> #
> # HD-Audio
> #
> CONFIG_SND_HDA=m
> CONFIG_SND_HDA_INTEL=m
> CONFIG_SND_HDA_HWDEP=y
> CONFIG_SND_HDA_RECONFIG=y
> CONFIG_SND_HDA_INPUT_BEEP=y
> CONFIG_SND_HDA_INPUT_BEEP_MODE=0
> CONFIG_SND_HDA_PATCH_LOADER=y
> CONFIG_SND_HDA_CODEC_REALTEK=m
> CONFIG_SND_HDA_CODEC_ANALOG=m
> CONFIG_SND_HDA_CODEC_SIGMATEL=m
> CONFIG_SND_HDA_CODEC_VIA=m
> CONFIG_SND_HDA_CODEC_HDMI=m
> CONFIG_SND_HDA_CODEC_CIRRUS=m
> CONFIG_SND_HDA_CODEC_CONEXANT=m
> CONFIG_SND_HDA_CODEC_CA0110=m
> CONFIG_SND_HDA_CODEC_CA0132=m
> # CONFIG_SND_HDA_CODEC_CA0132_DSP is not set
> CONFIG_SND_HDA_CODEC_CMEDIA=m
> CONFIG_SND_HDA_CODEC_SI3054=m
> CONFIG_SND_HDA_GENERIC=m
> CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
> CONFIG_SND_HDA_CORE=m
> CONFIG_SND_HDA_I915=y
> CONFIG_SND_HDA_PREALLOC_SIZE=4096
3937a3847,3852
> # CONFIG_SND_USB_HIFACE is not set
> # CONFIG_SND_BCD2000 is not set
> # CONFIG_SND_USB_POD is not set
> # CONFIG_SND_USB_PODHD is not set
> # CONFIG_SND_USB_TONEPORT is not set
> # CONFIG_SND_USB_VARIAX is not set
3940c3855,3856
< CONFIG_SND_FIREWIRE_SPEAKERS=m
---
> # CONFIG_SND_DICE is not set
> # CONFIG_SND_OXFW is not set
3941a3858,3862
> # CONFIG_SND_SCS1X is not set
> # CONFIG_SND_FIREWORKS is not set
> # CONFIG_SND_BEBOB is not set
> # CONFIG_SND_FIREWIRE_DIGI00X is not set
> # CONFIG_SND_FIREWIRE_TASCAM is not set
3946,3949d3866
< CONFIG_HID_SUPPORT=y
< CONFIG_HID=y
< # CONFIG_HID_BATTERY_STRENGTH is not set
< CONFIG_HIDRAW=y
3952c3869
< # USB Input Devices
---
> # HID support
3954,3956c3871,3875
< CONFIG_USB_HID=y
< CONFIG_HID_PID=y
< CONFIG_USB_HIDDEV=y
---
> CONFIG_HID=y
> # CONFIG_HID_BATTERY_STRENGTH is not set
> CONFIG_HIDRAW=y
> # CONFIG_UHID is not set
> CONFIG_HID_GENERIC=y
3964a3884,3885
> # CONFIG_HID_APPLEIR is not set
> # CONFIG_HID_AUREAL is not set
3965a3887
> # CONFIG_HID_BETOP_FF is not set
3967a3890
> # CONFIG_HID_CORSAIR is not set
3973a3897
> # CONFIG_HID_ELO is not set
3974a3899,3900
> # CONFIG_HID_GEMBIRD is not set
> # CONFIG_HID_GFRM is not set
3976a3903
> # CONFIG_HID_GT683R is not set
3981a3909
> # CONFIG_HID_ICADE is not set
3984a3913
> # CONFIG_HID_LENOVO is not set
3986a3916
> CONFIG_HID_LOGITECH_HIDPP=m
3998a3929
> # CONFIG_HID_PENMOUNT is not set
4004a3936
> # CONFIG_HID_PLANTRONICS is not set
4009a3942
> # CONFIG_SONY_FF is not set
4010a3944
> # CONFIG_HID_STEELSERIES is not set
4011a3946
> # CONFIG_HID_RMI is not set
4014d3948
< CONFIG_HID_HYPERV_MOUSE=m
4018a3953
> # CONFIG_HID_THINGM is not set
4022d3956
< CONFIG_HID_WACOM_POWER_SUPPLY=y
4024c3958
< CONFIG_HID_WIIMOTE_EXT=y
---
> # CONFIG_HID_XINMO is not set
4028,4030c3962,3975
< CONFIG_USB_ARCH_HAS_OHCI=y
< CONFIG_USB_ARCH_HAS_EHCI=y
< CONFIG_USB_ARCH_HAS_XHCI=y
---
> # CONFIG_HID_SENSOR_HUB is not set
> 
> #
> # USB HID support
> #
> CONFIG_USB_HID=y
> CONFIG_HID_PID=y
> CONFIG_USB_HIDDEV=y
> 
> #
> # I2C HID support
> #
> # CONFIG_I2C_HID is not set
> CONFIG_USB_OHCI_LITTLE_ENDIAN=y
4035d3979
< # CONFIG_USB_DEBUG is not set
4041,4042c3985
< CONFIG_USB_DEVICEFS=y
< # CONFIG_USB_DEVICE_CLASS is not set
---
> CONFIG_USB_DEFAULT_PERSIST=y
4044d3986
< CONFIG_USB_SUSPEND=y
4045a3988,3989
> # CONFIG_USB_OTG_WHITELIST is not set
> # CONFIG_USB_ULPI_BUS is not set
4056c4000,4001
< # CONFIG_USB_XHCI_HCD_DEBUGGING is not set
---
> CONFIG_USB_XHCI_PCI=y
> # CONFIG_USB_XHCI_PLATFORM is not set
4059a4005,4006
> CONFIG_USB_EHCI_PCI=y
> # CONFIG_USB_EHCI_HCD_PLATFORM is not set
4062d4008
< # CONFIG_USB_ISP1760_HCD is not set
4063a4010
> # CONFIG_USB_FOTG210_HCD is not set
4064a4012
> CONFIG_USB_OHCI_HCD_PCI=y
4066,4069d4013
< # CONFIG_USB_EHCI_HCD_PLATFORM is not set
< # CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
< # CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
< CONFIG_USB_OHCI_LITTLE_ENDIAN=y
4077a4022,4024
> # CONFIG_USB_HCD_BCMA is not set
> # CONFIG_USB_HCD_SSB is not set
> # CONFIG_USB_HCD_TEST_MODE is not set
4111d4057
< # CONFIG_USB_LIBUSUAL is not set
4117a4064,4069
> # CONFIG_USBIP_CORE is not set
> # CONFIG_USB_MUSB_HDRC is not set
> # CONFIG_USB_DWC3 is not set
> # CONFIG_USB_DWC2 is not set
> # CONFIG_USB_CHIPIDEA is not set
> # CONFIG_USB_ISP1760 is not set
4125d4076
< CONFIG_USB_EZUSB=y
4126a4078
> # CONFIG_USB_SERIAL_SIMPLE is not set
4137d4088
< CONFIG_USB_SERIAL_FUNSOFT=m
4156c4107
< CONFIG_USB_SERIAL_MOTOROLA=m
---
> # CONFIG_USB_SERIAL_MXUPORT is not set
4163d4113
< CONFIG_USB_SERIAL_HP4X=m
4166d4115
< CONFIG_USB_SERIAL_SIEMENS_MPI=m
4176,4177c4125,4126
< CONFIG_USB_SERIAL_VIVOPAY_SERIAL=m
< # CONFIG_USB_SERIAL_ZIO is not set
---
> # CONFIG_USB_SERIAL_XSENS_MT is not set
> # CONFIG_USB_SERIAL_WISHBONE is not set
4178a4128
> # CONFIG_USB_SERIAL_QT2 is not set
4202a4153
> # CONFIG_USB_EHSET_TEST_FIXTURE is not set
4204a4156,4159
> CONFIG_USB_EZUSB_FX2=m
> # CONFIG_USB_HSIC_USB3503 is not set
> # CONFIG_USB_LINK_LAYER_TEST is not set
> # CONFIG_USB_CHAOSKEY is not set
4210d4164
< # CONFIG_USB_GADGET is not set
4213c4167
< # OTG and related infrastructure
---
> # USB Physical Layer drivers
4215c4169
< CONFIG_USB_OTG_UTILS=y
---
> CONFIG_USB_PHY=y
4216a4171,4173
> # CONFIG_USB_ISP1301 is not set
> # CONFIG_USB_GADGET is not set
> # CONFIG_USB_LED_TRIG is not set
4223,4224d4179
< # CONFIG_MMC_UNSAFE_RESUME is not set
< # CONFIG_MMC_CLKGATE is not set
4240a4196
> # CONFIG_MMC_SDHCI_ACPI is not set
4248a4205,4207
> # CONFIG_MMC_USDHI6ROL0 is not set
> # CONFIG_MMC_TOSHIBA_PCI is not set
> # CONFIG_MMC_MTK is not set
4256a4216
> # CONFIG_MS_BLOCK is not set
4265a4226
> # CONFIG_LEDS_CLASS_FLASH is not set
4270a4232
> # CONFIG_LEDS_LM3642 is not set
4272a4235
> CONFIG_LEDS_LP55XX_COMMON=m
4274a4238,4240
> # CONFIG_LEDS_LP5562 is not set
> # CONFIG_LEDS_LP8501 is not set
> # CONFIG_LEDS_LP8860 is not set
4277c4243
< # CONFIG_LEDS_PCA9633 is not set
---
> # CONFIG_LEDS_PCA963X is not set
4282,4283c4248,4254
< # CONFIG_LEDS_OT200 is not set
< CONFIG_LEDS_TRIGGERS=y
---
> # CONFIG_LEDS_TLC591XX is not set
> # CONFIG_LEDS_LM355x is not set
> 
> #
> # LED driver for blink(1) USB RGB LED is under Special HID drivers (HID_THINGM)
> #
> # CONFIG_LEDS_BLINKM is not set
4287a4259
> CONFIG_LEDS_TRIGGERS=y
4288a4261
> # CONFIG_LEDS_TRIGGER_ONESHOT is not set
4290a4264
> # CONFIG_LEDS_TRIGGER_CPU is not set
4295a4270,4271
> # CONFIG_LEDS_TRIGGER_TRANSIENT is not set
> # CONFIG_LEDS_TRIGGER_CAMERA is not set
4301a4278
> CONFIG_INFINIBAND_ON_DEMAND_PAGING=y
4305d4281
< CONFIG_INFINIBAND_IPATH=m
4307,4308c4283
< CONFIG_INFINIBAND_AMSO1100=m
< # CONFIG_INFINIBAND_AMSO1100_DEBUG is not set
---
> CONFIG_INFINIBAND_QIB_DCA=y
4314a4290,4291
> # CONFIG_INFINIBAND_OCRDMA is not set
> # CONFIG_INFINIBAND_USNIC is not set
4321a4299,4301
> # CONFIG_INFINIBAND_ISERT is not set
> CONFIG_EDAC_ATOMIC_SCRUB=y
> CONFIG_EDAC_SUPPORT=y
4323,4326c4303
< 
< #
< # Reporting subsystems
< #
---
> CONFIG_EDAC_LEGACY_SYSFS=y
4329d4305
< CONFIG_EDAC_MCE_INJ=m
4336a4313
> # CONFIG_EDAC_IE31200 is not set
4347a4325,4326
> CONFIG_RTC_SYSTOHC=y
> CONFIG_RTC_SYSTOHC_DEVICE="rtc0"
4361a4341,4342
> # CONFIG_RTC_DRV_ABB5ZES3 is not set
> # CONFIG_RTC_DRV_ABX80X is not set
4363a4345
> # CONFIG_RTC_DRV_DS1374_WDT is not set
4369a4352
> # CONFIG_RTC_DRV_ISL12057 is not set
4370a4354,4355
> # CONFIG_RTC_DRV_PCF2127 is not set
> # CONFIG_RTC_DRV_PCF8523 is not set
4371a4357
> # CONFIG_RTC_DRV_PCF85063 is not set
4381a4368
> # CONFIG_RTC_DRV_RV8803 is not set
4393a4381
> # CONFIG_RTC_DRV_DS1685_FAMILY is not set
4394a4383
> # CONFIG_RTC_DRV_DS2404 is not set
4406a4396,4400
> 
> #
> # HID Sensor RTC drivers
> #
> # CONFIG_RTC_DRV_HID_SENSOR_TIME is not set
4413,4416d4406
< # CONFIG_INTEL_MID_DMAC is not set
< CONFIG_INTEL_IOATDMA=m
< # CONFIG_TIMB_DMA is not set
< CONFIG_PCH_DMA=m
4417a4408,4412
> CONFIG_DMA_ACPI=y
> # CONFIG_INTEL_IDMA64 is not set
> CONFIG_INTEL_IOATDMA=m
> # CONFIG_DW_DMAC is not set
> # CONFIG_DW_DMAC_PCI is not set
4422d4416
< CONFIG_NET_DMA=y
4424a4419
> CONFIG_DMA_ENGINE_RAID=y
4434d4428
< # CONFIG_UIO_PDRV is not set
4435a4430
> # CONFIG_UIO_DMEM_GENIRQ is not set
4439a4435,4439
> # CONFIG_UIO_PRUSS is not set
> # CONFIG_UIO_MF624 is not set
> # CONFIG_VFIO is not set
> CONFIG_IRQ_BYPASS_MANAGER=m
> # CONFIG_VIRT_DRIVERS is not set
4441d4440
< CONFIG_VIRTIO_RING=y
4446a4446
> CONFIG_VIRTIO_PCI_LEGACY=y
4447a4448
> # CONFIG_VIRTIO_INPUT is not set
4448a4450
> # CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES is not set
4453,4474d4454
< CONFIG_HYPERV=m
< CONFIG_HYPERV_UTILS=m
< 
< #
< # Xen driver support
< #
< CONFIG_XEN_BALLOON=y
< CONFIG_XEN_SELFBALLOONING=y
< CONFIG_XEN_SCRUB_PAGES=y
< CONFIG_XEN_DEV_EVTCHN=m
< CONFIG_XEN_BACKEND=y
< CONFIG_XENFS=m
< CONFIG_XEN_COMPAT_XENFS=y
< CONFIG_XEN_SYS_HYPERVISOR=y
< CONFIG_XEN_XENBUS_FRONTEND=y
< CONFIG_XEN_GNTDEV=m
< CONFIG_XEN_GRANT_DEV_ALLOC=m
< CONFIG_SWIOTLB_XEN=y
< CONFIG_XEN_TMEM=y
< CONFIG_XEN_PCIDEV_BACKEND=m
< CONFIG_XEN_PRIVCMD=m
< CONFIG_XEN_ACPI_PROCESSOR=m
4476d4455
< # CONFIG_ET131X is not set
4478,4479d4456
< # CONFIG_USBIP_CORE is not set
< # CONFIG_W35UND is not set
4481d4457
< # CONFIG_ECHO is not set
4483d4458
< # CONFIG_ASUS_OLED is not set
4485d4459
< # CONFIG_R8187SE is not set
4489,4495c4463,4465
< # CONFIG_RTS_PSTOR is not set
< # CONFIG_RTS5139 is not set
< # CONFIG_TRANZPORT is not set
< # CONFIG_IDE_PHISON is not set
< # CONFIG_LINE6_USB is not set
< # CONFIG_USB_SERIAL_QUATECH2 is not set
< # CONFIG_USB_SERIAL_QUATECH_USB2 is not set
---
> # CONFIG_R8188EU is not set
> # CONFIG_R8723AU is not set
> # CONFIG_RTS5208 is not set
4498,4507c4468
< # CONFIG_VME_BUS is not set
< # CONFIG_DX_SEP is not set
< # CONFIG_IIO is not set
< # CONFIG_ZRAM is not set
< # CONFIG_ZCACHE is not set
< # CONFIG_ZSMALLOC is not set
< # CONFIG_WLAGS49_H2 is not set
< # CONFIG_WLAGS49_H25 is not set
< # CONFIG_FB_SM7XX is not set
< CONFIG_CRYSTALHD=m
---
> # CONFIG_FB_SM750 is not set
4509,4512d4469
< # CONFIG_ACPI_QUICKSTART is not set
< # CONFIG_USB_ENESTORAGE is not set
< # CONFIG_BCM_WIMAX is not set
< # CONFIG_FT1000 is not set
4519d4475
< # CONFIG_INTEL_MEI is not set
4521,4537c4477
< # CONFIG_DVB_AS102 is not set
< # CONFIG_DVB_CXD2099 is not set
< # CONFIG_VIDEO_DT3155 is not set
< # CONFIG_EASYCAP is not set
< # CONFIG_VIDEO_GO7007 is not set
< # CONFIG_SOLO6X10 is not set
< CONFIG_LIRC_STAGING=y
< CONFIG_LIRC_BT829=m
< CONFIG_LIRC_IGORPLUGUSB=m
< CONFIG_LIRC_IMON=m
< CONFIG_LIRC_PARALLEL=m
< CONFIG_LIRC_SASEM=m
< CONFIG_LIRC_SERIAL=m
< CONFIG_LIRC_SERIAL_TRANSMITTER=y
< CONFIG_LIRC_SIR=m
< CONFIG_LIRC_TTUSBIR=m
< CONFIG_LIRC_ZILOG=m
---
> # CONFIG_STAGING_RDMA is not set
4542,4545c4482,4492
< # CONFIG_ANDROID is not set
< # CONFIG_PHONE is not set
< # CONFIG_RAMSTER is not set
< # CONFIG_USB_WPAN_HCD is not set
---
> # CONFIG_WIMAX_GDM72XX is not set
> # CONFIG_LTE_GDM724X is not set
> # CONFIG_FIREWIRE_SERIAL is not set
> # CONFIG_LUSTRE_FS is not set
> # CONFIG_DGNC is not set
> # CONFIG_DGAP is not set
> # CONFIG_GS_FPGABOOT is not set
> # CONFIG_CRYPTO_SKEIN is not set
> # CONFIG_UNISYSSPAR is not set
> # CONFIG_WILC1000_DRIVER is not set
> # CONFIG_MOST is not set
4548a4496
> # CONFIG_ALIENWARE_WMI is not set
4552a4501,4502
> # CONFIG_DELL_SMO8800 is not set
> # CONFIG_DELL_RBTN is not set
4557a4508
> # CONFIG_HP_WIRELESS is not set
4582a4534,4535
> # CONFIG_TOSHIBA_HAPS is not set
> # CONFIG_TOSHIBA_WMI is not set
4586d4538
< # CONFIG_XO15_EBOOK is not set
4591a4544,4549
> # CONFIG_INTEL_RST is not set
> # CONFIG_INTEL_SMARTCONNECT is not set
> # CONFIG_PVPANIC is not set
> # CONFIG_INTEL_PMC_IPC is not set
> # CONFIG_SURFACE_PRO3_BUTTON is not set
> # CONFIG_CHROME_PLATFORMS is not set
4595a4554,4557
> 
> #
> # Clock Source drivers
> #
4598a4561,4566
> # CONFIG_ATMEL_PIT is not set
> # CONFIG_SH_TIMER_CMT is not set
> # CONFIG_SH_TIMER_MTU2 is not set
> # CONFIG_SH_TIMER_TMU is not set
> # CONFIG_EM_TIMER_STI is not set
> # CONFIG_MAILBOX is not set
4600a4569,4573
> 
> #
> # Generic IOMMU Pagetable Support
> #
> CONFIG_IOMMU_IOVA=y
4605a4579
> # CONFIG_INTEL_IOMMU_SVM is not set
4611c4585
< # Remoteproc drivers (EXPERIMENTAL)
---
> # Remoteproc drivers
4612a4587
> # CONFIG_STE_MODEM_RPROC is not set
4615c4590
< # Rpmsg drivers (EXPERIMENTAL)
---
> # Rpmsg drivers
4617,4618c4592,4653
< # CONFIG_VIRT_DRIVERS is not set
< # CONFIG_PM_DEVFREQ is not set
---
> 
> #
> # SOC (System On Chip) specific Drivers
> #
> # CONFIG_SUNXI_SRAM is not set
> # CONFIG_SOC_TI is not set
> CONFIG_PM_DEVFREQ=y
> 
> #
> # DEVFREQ Governors
> #
> CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=m
> # CONFIG_DEVFREQ_GOV_PERFORMANCE is not set
> # CONFIG_DEVFREQ_GOV_POWERSAVE is not set
> # CONFIG_DEVFREQ_GOV_USERSPACE is not set
> 
> #
> # DEVFREQ Drivers
> #
> # CONFIG_PM_DEVFREQ_EVENT is not set
> # CONFIG_EXTCON is not set
> # CONFIG_MEMORY is not set
> # CONFIG_IIO is not set
> # CONFIG_NTB is not set
> # CONFIG_VME_BUS is not set
> # CONFIG_PWM is not set
> # CONFIG_IPACK_BUS is not set
> CONFIG_RESET_CONTROLLER=y
> # CONFIG_FMC is not set
> 
> #
> # PHY Subsystem
> #
> CONFIG_GENERIC_PHY=y
> # CONFIG_PHY_PXA_28NM_HSIC is not set
> # CONFIG_PHY_PXA_28NM_USB2 is not set
> # CONFIG_BCM_KONA_USB2_PHY is not set
> # CONFIG_POWERCAP is not set
> # CONFIG_MCB is not set
> 
> #
> # Performance monitor support
> #
> CONFIG_RAS=y
> # CONFIG_AMD_MCE_INJ is not set
> # CONFIG_THUNDERBOLT is not set
> 
> #
> # Android
> #
> # CONFIG_ANDROID is not set
> # CONFIG_LIBNVDIMM is not set
> # CONFIG_NVMEM is not set
> # CONFIG_STM is not set
> # CONFIG_STM_DUMMY is not set
> # CONFIG_STM_SOURCE_CONSOLE is not set
> # CONFIG_INTEL_TH is not set
> 
> #
> # FPGA Configuration Support
> #
> # CONFIG_FPGA is not set
4626d4660
< CONFIG_EFI_VARS=y
4630a4665
> CONFIG_DMI_SCAN_MACHINE_NON_EFI_FALLBACK=y
4635a4671,4682
> # EFI (Extensible Firmware Interface) Support
> #
> CONFIG_EFI_VARS=y
> CONFIG_EFI_ESRT=y
> CONFIG_EFI_VARS_PSTORE=y
> # CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE is not set
> CONFIG_EFI_RUNTIME_MAP=y
> # CONFIG_EFI_FAKE_MEMMAP is not set
> CONFIG_EFI_RUNTIME_WRAPPERS=y
> CONFIG_UEFI_CPER=y
> 
> #
4642,4643c4689
< CONFIG_EXT4_USE_FOR_EXT23=y
< CONFIG_EXT4_FS_XATTR=y
---
> CONFIG_EXT4_USE_FOR_EXT2=y
4645a4692
> # CONFIG_EXT4_ENCRYPTION is not set
4664a4712
> # CONFIG_XFS_WARN is not set
4676a4725,4727
> # CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
> # CONFIG_BTRFS_DEBUG is not set
> # CONFIG_BTRFS_ASSERT is not set
4677a4729,4730
> # CONFIG_F2FS_FS is not set
> # CONFIG_FS_DAX is not set
4698c4751
< CONFIG_GENERIC_ACL=y
---
> # CONFIG_OVERLAY_FS is not set
4738a4792,4793
> # CONFIG_PROC_CHILDREN is not set
> CONFIG_KERNFS=y
4745a4801
> CONFIG_EFIVAR_FS=m
4749a4806
> # CONFIG_ECRYPT_FS_MESSAGING is not set
4751a4809
> # CONFIG_HFSPLUS_FS_POSIX_ACL is not set
4760a4819,4823
> CONFIG_SQUASHFS_FILE_CACHE=y
> # CONFIG_SQUASHFS_FILE_DIRECT is not set
> CONFIG_SQUASHFS_DECOMP_SINGLE=y
> # CONFIG_SQUASHFS_DECOMP_MULTI is not set
> # CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU is not set
4762a4826
> # CONFIG_SQUASHFS_LZ4 is not set
4779a4844,4847
> # CONFIG_PSTORE_CONSOLE is not set
> # CONFIG_PSTORE_PMSG is not set
> # CONFIG_PSTORE_FTRACE is not set
> # CONFIG_PSTORE_RAM is not set
4788c4856,4857
< CONFIG_NFS_V3=y
---
> CONFIG_NFS_V2=m
> CONFIG_NFS_V3=m
4790c4859,4860
< CONFIG_NFS_V4=y
---
> CONFIG_NFS_V4=m
> # CONFIG_NFS_SWAP is not set
4791a4862
> # CONFIG_NFS_V4_2 is not set
4794a4866
> CONFIG_PNFS_FLEXFILE_LAYOUT=m
4795a4868
> # CONFIG_NFS_V4_1_MIGRATION is not set
4804a4878,4879
> # CONFIG_NFSD_PNFS is not set
> # CONFIG_NFSD_V4_SECURITY_LABEL is not set
4805a4881
> CONFIG_GRACE_PERIOD=m
4813d4888
< CONFIG_SUNRPC_XPRT_RDMA=m
4815a4891
> CONFIG_SUNRPC_XPRT_RDMA=m
4816a4893,4894
> # CONFIG_CEPH_FSCACHE is not set
> # CONFIG_CEPH_FS_POSIX_ACL is not set
4823a4902,4903
> CONFIG_CIFS_ACL=y
> CONFIG_CIFS_DEBUG=y
4825a4906
> # CONFIG_CIFS_SMB2 is not set
4827d4907
< CONFIG_CIFS_ACL=y
4841a4922
> # CONFIG_9P_FS_SECURITY is not set
4880a4962,4972
> # CONFIG_NLS_MAC_ROMAN is not set
> # CONFIG_NLS_MAC_CELTIC is not set
> # CONFIG_NLS_MAC_CENTEURO is not set
> # CONFIG_NLS_MAC_CROATIAN is not set
> # CONFIG_NLS_MAC_CYRILLIC is not set
> # CONFIG_NLS_MAC_GAELIC is not set
> # CONFIG_NLS_MAC_GREEK is not set
> # CONFIG_NLS_MAC_ICELAND is not set
> # CONFIG_NLS_MAC_INUIT is not set
> # CONFIG_NLS_MAC_ROMANIAN is not set
> # CONFIG_NLS_MAC_TURKISH is not set
4888a4981,4984
> 
> #
> # printk and dmesg options
> #
4890c4986,4997
< CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
---
> CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4
> CONFIG_BOOT_PRINTK_DELAY=y
> CONFIG_DYNAMIC_DEBUG=y
> 
> #
> # Compile-time checks and compiler options
> #
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> # CONFIG_GDB_SCRIPTS is not set
4894d5000
< CONFIG_MAGIC_SYSRQ=y
4895a5002
> # CONFIG_READABLE_ASM is not set
4896a5004
> # CONFIG_PAGE_OWNER is not set
4899a5008,5013
> CONFIG_SECTION_MISMATCH_WARN_ONLY=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_FRAME_POINTER=y
> # CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
4900a5015,5037
> 
> #
> # Memory Debugging
> #
> # CONFIG_PAGE_EXTENSION is not set
> # CONFIG_DEBUG_PAGEALLOC is not set
> # CONFIG_DEBUG_OBJECTS is not set
> # CONFIG_SLUB_DEBUG_ON is not set
> # CONFIG_SLUB_STATS is not set
> CONFIG_HAVE_DEBUG_KMEMLEAK=y
> # CONFIG_DEBUG_KMEMLEAK is not set
> # CONFIG_DEBUG_STACK_USAGE is not set
> CONFIG_DEBUG_VM=y
> # CONFIG_DEBUG_VM_VMACACHE is not set
> # CONFIG_DEBUG_VM_RB is not set
> # CONFIG_DEBUG_VIRTUAL is not set
> CONFIG_DEBUG_MEMORY_INIT=y
> # CONFIG_DEBUG_PER_CPU_MAPS is not set
> CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
> CONFIG_DEBUG_STACKOVERFLOW=y
> CONFIG_HAVE_ARCH_KMEMCHECK=y
> CONFIG_HAVE_ARCH_KASAN=y
> # CONFIG_KASAN is not set
4901a5039,5042
> 
> #
> # Debug Lockups and Hangs
> #
4908a5050,5052
> # CONFIG_PANIC_ON_OOPS is not set
> CONFIG_PANIC_ON_OOPS_VALUE=0
> CONFIG_PANIC_TIMEOUT=0
4909a5054
> CONFIG_SCHED_INFO=y
4910a5056,5057
> # CONFIG_SCHED_STACK_END_CHECK is not set
> # CONFIG_DEBUG_TIMEKEEPING is not set
4912,4915c5059,5062
< # CONFIG_DEBUG_OBJECTS is not set
< # CONFIG_SLUB_DEBUG_ON is not set
< # CONFIG_SLUB_STATS is not set
< # CONFIG_DEBUG_KMEMLEAK is not set
---
> 
> #
> # Lock Debugging (spinlocks, mutexes, etc...)
> #
4917d5063
< # CONFIG_RT_MUTEX_TESTER is not set
4919a5066
> # CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
4922d5068
< CONFIG_SPARSE_RCU_POINTER=y
4925a5072
> # CONFIG_LOCK_TORTURE_TEST is not set
4927d5073
< # CONFIG_DEBUG_STACK_USAGE is not set
4930,4935d5075
< CONFIG_DEBUG_INFO=y
< # CONFIG_DEBUG_INFO_REDUCED is not set
< CONFIG_DEBUG_VM=y
< # CONFIG_DEBUG_VIRTUAL is not set
< # CONFIG_DEBUG_WRITECOUNT is not set
< CONFIG_DEBUG_MEMORY_INIT=y
4937c5077
< # CONFIG_TEST_LIST_SORT is not set
---
> # CONFIG_DEBUG_PI_LIST is not set
4941,4943c5081,5087
< CONFIG_ARCH_WANT_FRAME_POINTERS=y
< CONFIG_FRAME_POINTER=y
< CONFIG_BOOT_PRINTK_DELAY=y
---
> 
> #
> # RCU Debugging
> #
> # CONFIG_PROVE_RCU is not set
> CONFIG_SPARSE_RCU_POINTER=y
> # CONFIG_TORTURE_TEST is not set
4946d5089
< # CONFIG_RCU_CPU_STALL_INFO is not set
4948,4949c5091
< # CONFIG_KPROBES_SANITY_TEST is not set
< # CONFIG_BACKTRACE_SELF_TEST is not set
---
> # CONFIG_RCU_EQS_DEBUG is not set
4951,4954c5093
< # CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
< # CONFIG_DEBUG_PER_CPU_MAPS is not set
< # CONFIG_LKDTM is not set
< # CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set
---
> # CONFIG_NOTIFIER_ERROR_INJECTION is not set
4957c5096,5097
< # CONFIG_DEBUG_PAGEALLOC is not set
---
> CONFIG_ARCH_HAS_DEBUG_STRICT_USER_COPY_CHECKS=y
> # CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set
4960d5099
< CONFIG_HAVE_FTRACE_NMI_ENTER=y
4964d5102
< CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
4965a5104
> CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
4967a5107
> CONFIG_HAVE_FENTRY=y
4969a5110
> CONFIG_TRACE_CLOCK=y
4971d5111
< CONFIG_FTRACE_NMI_ENTER=y
4973d5112
< CONFIG_EVENT_POWER_TRACING_DEPRECATED=y
4984a5124,5125
> CONFIG_TRACER_SNAPSHOT=y
> # CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP is not set
4990a5132,5133
> # CONFIG_UPROBE_EVENT is not set
> CONFIG_PROBE_EVENTS=y
4991a5135
> CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
4995a5140
> # CONFIG_TRACEPOINT_BENCHMARK is not set
4996a5142,5161
> # CONFIG_RING_BUFFER_STARTUP_TEST is not set
> # CONFIG_TRACE_ENUM_MAP_FILE is not set
> 
> #
> # Runtime Testing
> #
> # CONFIG_LKDTM is not set
> # CONFIG_TEST_LIST_SORT is not set
> # CONFIG_KPROBES_SANITY_TEST is not set
> # CONFIG_BACKTRACE_SELF_TEST is not set
> # CONFIG_RBTREE_TEST is not set
> # CONFIG_INTERVAL_TREE_TEST is not set
> # CONFIG_PERCPU_TEST is not set
> CONFIG_ATOMIC64_SELFTEST=y
> CONFIG_ASYNC_RAID6_TEST=m
> # CONFIG_TEST_HEXDUMP is not set
> # CONFIG_TEST_STRING_HELPERS is not set
> CONFIG_TEST_KSTRTOX=y
> # CONFIG_TEST_PRINTF is not set
> # CONFIG_TEST_RHASHTABLE is not set
4998d5162
< # CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set
5000d5163
< CONFIG_DYNAMIC_DEBUG=y
5002,5003c5165,5171
< CONFIG_ATOMIC64_SELFTEST=y
< CONFIG_ASYNC_RAID6_TEST=m
---
> # CONFIG_TEST_LKM is not set
> # CONFIG_TEST_USER_COPY is not set
> # CONFIG_TEST_BPF is not set
> # CONFIG_TEST_FIRMWARE is not set
> # CONFIG_TEST_UDELAY is not set
> # CONFIG_MEMTEST is not set
> # CONFIG_TEST_STATIC_KEYS is not set
5011a5180
> CONFIG_KDB_DEFAULT_ENABLE=0x1
5013,5014c5182
< CONFIG_HAVE_ARCH_KMEMCHECK=y
< CONFIG_TEST_KSTRTOX=y
---
> CONFIG_KDB_CONTINUE_CATASTROPHIC=0
5019c5187,5188
< CONFIG_DEBUG_STACKOVERFLOW=y
---
> # CONFIG_EARLY_PRINTK_EFI is not set
> # CONFIG_X86_PTDUMP_CORE is not set
5020a5190
> # CONFIG_EFI_PGT_DUMP is not set
5022a5193
> # CONFIG_DEBUG_WX is not set
5024a5196,5197
> CONFIG_DOUBLEFAULT=y
> # CONFIG_DEBUG_TLBFLUSH is not set
5041c5214
< # CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set
---
> # CONFIG_DEBUG_ENTRY is not set
5042a5216,5218
> # CONFIG_X86_DEBUG_STATIC_CPU_HAS is not set
> CONFIG_X86_DEBUG_FPU=y
> # CONFIG_PUNIT_ATOM_DEBUG is not set
5047a5224,5225
> # CONFIG_PERSISTENT_KEYRINGS is not set
> # CONFIG_BIG_KEYS is not set
5050d5227
< CONFIG_KEYS_DEBUG_PROC_KEYS=y
5070a5248,5250
> CONFIG_INTEGRITY=y
> # CONFIG_INTEGRITY_SIGNATURE is not set
> CONFIG_INTEGRITY_AUDIT=y
5071a5252
> # CONFIG_EVM is not set
5081,5082d5261
< CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
< CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
5088d5266
< CONFIG_CRYPTO_FIPS=y
5098a5277
> CONFIG_CRYPTO_RNG_DEFAULT=y
5100a5280,5281
> CONFIG_CRYPTO_AKCIPHER2=y
> # CONFIG_CRYPTO_RSA is not set
5105,5106c5286,5288
< CONFIG_CRYPTO_GF128MUL=m
< CONFIG_CRYPTO_NULL=m
---
> CONFIG_CRYPTO_GF128MUL=y
> CONFIG_CRYPTO_NULL=y
> CONFIG_CRYPTO_NULL2=y
5109a5292
> # CONFIG_CRYPTO_MCRYPTD is not set
5111a5295,5296
> CONFIG_CRYPTO_ABLK_HELPER=y
> CONFIG_CRYPTO_GLUE_HELPER_X86=y
5117a5303
> # CONFIG_CRYPTO_CHACHA20POLY1305 is not set
5118a5305
> CONFIG_CRYPTO_ECHAINIV=m
5127c5314
< CONFIG_CRYPTO_LRW=m
---
> CONFIG_CRYPTO_LRW=y
5129c5316,5317
< CONFIG_CRYPTO_XTS=m
---
> CONFIG_CRYPTO_XTS=y
> # CONFIG_CRYPTO_KEYWRAP is not set
5133a5322
> CONFIG_CRYPTO_CMAC=m
5142a5332,5335
> # CONFIG_CRYPTO_CRC32 is not set
> # CONFIG_CRYPTO_CRC32_PCLMUL is not set
> CONFIG_CRYPTO_CRCT10DIF=y
> # CONFIG_CRYPTO_CRCT10DIF_PCLMUL is not set
5143a5337,5338
> # CONFIG_CRYPTO_POLY1305 is not set
> # CONFIG_CRYPTO_POLY1305_X86_64 is not set
5153c5348,5351
< CONFIG_CRYPTO_SHA256=m
---
> # CONFIG_CRYPTO_SHA256_SSSE3 is not set
> # CONFIG_CRYPTO_SHA512_SSSE3 is not set
> # CONFIG_CRYPTO_SHA1_MB is not set
> CONFIG_CRYPTO_SHA256=y
5171a5370,5372
> # CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64 is not set
> # CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64 is not set
> CONFIG_CRYPTO_CAST_COMMON=m
5172a5374
> # CONFIG_CRYPTO_CAST5_AVX_X86_64 is not set
5173a5376
> # CONFIG_CRYPTO_CAST6_AVX_X86_64 is not set
5174a5378
> # CONFIG_CRYPTO_DES3_EDE_X86_64 is not set
5178a5383,5384
> # CONFIG_CRYPTO_CHACHA20 is not set
> # CONFIG_CRYPTO_CHACHA20_X86_64 is not set
5181a5388,5389
> # CONFIG_CRYPTO_SERPENT_AVX_X86_64 is not set
> # CONFIG_CRYPTO_SERPENT_AVX2_X86_64 is not set
5186a5395
> # CONFIG_CRYPTO_TWOFISH_AVX_X86_64 is not set
5193a5403,5405
> # CONFIG_CRYPTO_842 is not set
> # CONFIG_CRYPTO_LZ4 is not set
> # CONFIG_CRYPTO_LZ4HC is not set
5198a5411,5416
> CONFIG_CRYPTO_DRBG_MENU=y
> CONFIG_CRYPTO_DRBG_HMAC=y
> # CONFIG_CRYPTO_DRBG_HASH is not set
> # CONFIG_CRYPTO_DRBG_CTR is not set
> CONFIG_CRYPTO_DRBG=y
> CONFIG_CRYPTO_JITTERENTROPY=y
5201a5420,5421
> # CONFIG_CRYPTO_USER_API_RNG is not set
> # CONFIG_CRYPTO_USER_API_AEAD is not set
5205a5426,5434
> # CONFIG_CRYPTO_DEV_CCP is not set
> # CONFIG_CRYPTO_DEV_QAT_DH895xCC is not set
> # CONFIG_CRYPTO_DEV_QAT_DH895xCCVF is not set
> # CONFIG_ASYMMETRIC_KEY_TYPE is not set
> 
> #
> # Certificates for signature checking
> #
> # CONFIG_SYSTEM_TRUSTED_KEYRING is not set
5207a5437,5438
> CONFIG_HAVE_KVM_IRQFD=y
> CONFIG_HAVE_KVM_IRQ_ROUTING=y
5211a5443,5448
> CONFIG_HAVE_KVM_MSI=y
> CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
> CONFIG_KVM_VFIO=y
> CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
> CONFIG_KVM_COMPAT=y
> CONFIG_HAVE_KVM_IRQ_BYPASS=y
5217c5454
< CONFIG_VHOST_NET=m
---
> # CONFIG_KVM_DEVICE_ASSIGNMENT is not set
5224a5462,5466
> # CONFIG_HAVE_ARCH_BITREVERSE is not set
> CONFIG_RATIONAL=y
> CONFIG_GENERIC_STRNCPY_FROM_USER=y
> CONFIG_GENERIC_STRNLEN_USER=y
> CONFIG_GENERIC_NET_UTILS=y
5228a5471,5472
> CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
> CONFIG_ARCH_HAS_FAST_MULTIPLIER=y
5241a5486,5487
> # CONFIG_AUDIT_ARCH_COMPAT_GENERIC is not set
> # CONFIG_RANDOM32_SELFTEST is not set
5243c5489
< CONFIG_ZLIB_DEFLATE=m
---
> CONFIG_ZLIB_DEFLATE=y
5245a5492
> CONFIG_LZ4_DECOMPRESS=y
5259a5507
> CONFIG_DECOMPRESS_LZ4=y
5266a5515,5516
> CONFIG_INTERVAL_TREE=y
> CONFIG_ASSOCIATIVE_ARRAY=y
5268c5518
< CONFIG_HAS_IOPORT=y
---
> CONFIG_HAS_IOPORT_MAP=y
5272a5523,5524
> CONFIG_GLOB=y
> # CONFIG_GLOB_SELFTEST is not set
5273a5526
> CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y
5275d5527
< CONFIG_AVERAGE=y
5276a5529,5539
> # CONFIG_DDR is not set
> CONFIG_OID_REGISTRY=m
> CONFIG_UCS2_STRING=y
> CONFIG_FONT_SUPPORT=y
> # CONFIG_FONTS is not set
> CONFIG_FONT_8x8=y
> CONFIG_FONT_8x16=y
> # CONFIG_SG_SPLIT is not set
> CONFIG_ARCH_HAS_SG_CHAIN=y
> CONFIG_ARCH_HAS_PMEM_API=y
> CONFIG_ARCH_HAS_MMIO_FLUSH=y



^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Poorer networking performance in later kernels?
  2016-04-19 14:54         ` Butler, Peter
  2016-04-19 15:13           ` Josh Hunt
@ 2016-04-19 15:58           ` Oliver Hartkopp
  2016-04-19 16:00             ` Butler, Peter
  2016-04-19 16:13             ` David Miller
  1 sibling, 2 replies; 16+ messages in thread
From: Oliver Hartkopp @ 2016-04-19 15:58 UTC (permalink / raw)
  To: Butler, Peter, Rick Jones, netdev@vger.kernel.org

On 04/19/2016 04:54 PM, Butler, Peter wrote:

>
> I think the issue is resolved.  I had to recompile my 4.4.0 kernel with a few options pertaining to the Intel NIC which somehow (?) got left out or otherwise clobbered when I ported my 3.4.2 .config to the 4.4.0 kernel source tree.  With those changes now in I see essentially identical performance with the two kernels.  Sorry for any confusion and/or waste of time here.  My bad.
>

Can you please send the relevant changes in the config that caused the 
discussed issue?

Just in the case other people do a similar kernel upgrade from 3.x to a 
recent kernel and the current defaults lead to this non-optimal result.

Thanks,
Oliver

^ permalink raw reply	[flat|nested] 16+ messages in thread

* RE: Poorer networking performance in later kernels?
  2016-04-19 15:58           ` Oliver Hartkopp
@ 2016-04-19 16:00             ` Butler, Peter
  2016-04-19 16:13             ` David Miller
  1 sibling, 0 replies; 16+ messages in thread
From: Butler, Peter @ 2016-04-19 16:00 UTC (permalink / raw)
  To: Oliver Hartkopp, Rick Jones, netdev@vger.kernel.org

> I think the issue is resolved.  I had to recompile my 4.4.0 kernel with a few options pertaining to the Intel NIC which somehow (?) got left out or otherwise clobbered when I ported my 3.4.2 .config to the 4.4.0 kernel source tree.  With those changes now in I see essentially identical performance with the two kernels.  Sorry for any confusion and/or waste of time here.  My bad.
>

Can you please send the relevant changes in the config that caused the discussed issue?

Just in the case other people do a similar kernel upgrade from 3.x to a recent kernel and the current defaults lead to this non-optimal result.

Thanks,
Oliver


Yes I just replied to another email on this thread asking the same thing and posted some info.

Cheers,

Peter

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: Poorer networking performance in later kernels?
  2016-04-19 15:58           ` Oliver Hartkopp
  2016-04-19 16:00             ` Butler, Peter
@ 2016-04-19 16:13             ` David Miller
  1 sibling, 0 replies; 16+ messages in thread
From: David Miller @ 2016-04-19 16:13 UTC (permalink / raw)
  To: socketcan; +Cc: pbutler, rick.jones2, netdev

From: Oliver Hartkopp <socketcan@hartkopp.net>
Date: Tue, 19 Apr 2016 17:58:03 +0200

> On 04/19/2016 04:54 PM, Butler, Peter wrote:
> 
>>
>> I think the issue is resolved.  I had to recompile my 4.4.0 kernel
>> with a few options pertaining to the Intel NIC which somehow (?) got
>> left out or otherwise clobbered when I ported my 3.4.2 .config to the
>> 4.4.0 kernel source tree.  With those changes now in I see essentially
>> identical performance with the two kernels.  Sorry for any confusion
>> and/or waste of time here.  My bad.
>>
> 
> Can you please send the relevant changes in the config that caused the
> discussed issue?
> 
> Just in the case other people do a similar kernel upgrade from 3.x to
> a recent kernel and the current defaults lead to this non-optimal
> result.

+1

^ permalink raw reply	[flat|nested] 16+ messages in thread

* RE: Poorer networking performance in later kernels?
  2016-04-15 22:37   ` Rick Jones
  2016-04-18 11:27     ` Butler, Peter
@ 2016-04-18 12:02     ` Butler, Peter
  1 sibling, 0 replies; 16+ messages in thread
From: Butler, Peter @ 2016-04-18 12:02 UTC (permalink / raw)
  To: Rick Jones, netdev@vger.kernel.org

Just a minor clarification to my last paragraph ("When I perform the tests in this setup the 3.4.2 and 4.4.0 kernels perform identically - just as you would expect.").  By this I don't mean that the 3.4.2 and 4.4.0 kernels on the VMs perform identically to the 3.4.2 and 4.4.0 kernels on the actual hardware; what I mean is that in VM-land the original problem is essentially gone, as I get the same throughput with either kernel .

-----Original Message-----
From: Butler, Peter 
Sent: April-18-16 7:28 AM
To: 'Rick Jones' <rick.jones2@hpe.com>; netdev@vger.kernel.org
Subject: RE: Poorer networking performance in later kernels?

Hi Rick

Thanks for the reply.

Here is some hardware information, as requested (the two systems are identical, and are communicating with one another over a 10GB full-duplex Ethernet backplane):

- processor type: Intel(R) Xeon(R) CPU C5528  @ 2.13GHz
- NIC: Intel 82599EB 10GB XAUI/BX4
- NIC driver: ixgbe version 4.2.1-k (part of 4.4.0 kernel)

As for the buffer sizes, those rather large ones work fine for us with the 3.4.2 kernel.  However, for the sake of being complete, I have re-tried the tests with the 'standard' 4.4.0 kernel parameters for all /proc/sys/net/* values, and the results still were extremely poor in comparison to the 3.4.2 kernel.

Our MTU is actually just the standard 1500 bytes, however the message size was chosen to mimic actual traffic which will be segmented.

I ran ethtool -k (indeed I checked all ethtool parameters, not just those via -k) and the only real difference I could find was in "large-receive-offload" which was ON in 3.4.2 but OFF in 4.4.0 - so I used ethtool to change this to match the 3.4.2 settings and re-ran the tests.  Didn't help :-(   It's possible of course that I have missed a parameter here or there in comparing the 3.4.2 setup to the 4.4.0 setup.  I also tried running the ethtool config with the latest and greatest ethtool version (4.5) on the 4.4.0 kernel, as compared to the old 3.1 version on our 3.4.2 kernel.

I performed the TCP_RR test as requested and in that case, the results are much more comparable.  The old kernel is still better, but now only around 10% better as opposed to 2-3x better.

However I still contend that the *_STREAM tests are giving us more pertinent data, since our product application is only getting 1/3 to 1/2 half of the performance on the 4.4.0 kernel, and this is the same thing I see when I use netperf to test.

One other note: I tried running our 3.4.2 and 4.4.0 kernels in a VM environment on my workstation, so as to take the 'real' production hardware out of the equation.  When I perform the tests in this setup the 3.4.2 and 4.4.0 kernels perform identically - just as you would expect.

Any other ideas?  What can I be missing here?

Peter

-----Original Message-----
From: Rick Jones [mailto:rick.jones2@hpe.com]
Sent: April-15-16 6:37 PM
To: Butler, Peter <pbutler@sonusnet.com>; netdev@vger.kernel.org
Subject: Re: Poorer networking performance in later kernels?

On 04/15/2016 02:02 PM, Butler, Peter wrote:
> (Please keep me CC'd to all comments/responses)
>
> I've tried a kernel upgrade from 3.4.2 to 4.4.0 and see a marked drop 
> in networking performance.  Nothing was changed on the test systems, 
> other than the kernel itself (and kernel modules).  The identical 
> .config used to build the 3.4.2 kernel was brought over into the
> 4.4.0 kernel source tree, and any configuration differences (e.g. new 
> parameters, etc.) were taken as default values.
>
> The testing was performed on the same actual hardware for both kernel 
> versions (i.e. take the existing 3.4.2 physical setup, simply boot 
> into the (new) kernel and run the same test).  The netperf utility was 
> used for benchmarking and the testing was always performed on idle 
> systems.
>
> TCP testing yielded the following results, where the 4.4.0 kernel only 
> got about 1/2 of the throughput:
>

>        Recv     Send       Send                          Utilization       Service Demand
>        Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>        Size     Size       Size    Time       Throughput local    remote   local   remote
>        bytes    bytes      bytes   secs.      10^6bits/s % S      % S      us/KB   us/KB
>
> 3.4.2 13631488 13631488   8952    30.01      9370.29    10.14    6.50     0.709   0.454
> 4.4.0 13631488 13631488   8952    30.02      5314.03    9.14     14.31    1.127   1.765
>
> SCTP testing yielded the following results, where the 4.4.0 kernel only got about 1/3 of the throughput:
>
>        Recv     Send       Send                          Utilization       Service Demand
>        Socket   Socket     Message Elapsed               Send     Recv     Send    Recv
>        Size     Size       Size    Time       Throughput local    remote   local   remote
>        bytes    bytes      bytes   secs.      10^6bits/s  % S     % S      us/KB   us/KB
>
> 3.4.2 13631488 13631488   8952    30.00      2306.22    13.87    13.19    3.941   3.747
> 4.4.0 13631488 13631488   8952    30.01       882.74    16.86    19.14    12.516  14.210
>
> The same tests were performed a multitude of time, and are always 
> consistent (within a few percent).  I've also tried playing with 
> various run-time kernel parameters (/proc/sys/kernel/net/...) on the
> 4.4.0 kernel to alleviate the issue but have had no success at all.
>
> I'm at a loss as to what could possibly account for such a discrepancy...
>

I suspect I am not alone in being curious about the CPU(s) present in the systems and the model/whatnot of the NIC being used.  I'm also curious as to why you have what at first glance seem like absurdly large socket buffer sizes.

That said, it looks like you have some Really Big (tm) increases in service demand.  Many more CPU cycles being consumed per KB of data transferred.

Your message size makes me wonder if you were using a 9000 byte MTU.

Perhaps in the move from 3.4.2 to 4.4.0 you lost some or all of the stateless offloads for your NIC(s)?  Running ethtool -k <interface> on both ends under both kernels might be good.

Also, if you did have a 9000 byte MTU under 3.4.2 are you certain you still had it under 4.4.0?

It would (at least to me) also be interesting to run a TCP_RR test comparing the two kernels.  TCP_RR (at least with the default request/response size of one byte) doesn't really care about stateless offloads or MTUs and could show how much difference there is in basic path length (or I suppose in interrupt coalescing behaviour if the NIC in question has a mildly dodgy heuristic for such things).

happy benchmarking,

rick jones

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2016-04-19 16:21 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <SN1PR0301MB19987565B71A4688593A2CDBD6680@SN1PR0301MB1998.namprd03.prod.outlook.com>
2016-04-15 21:02 ` Poorer networking performance in later kernels? Butler, Peter
2016-04-15 22:33   ` Eric Dumazet
2016-04-18 12:16     ` Eric Dumazet
2016-04-18 16:27       ` Butler, Peter
2016-04-18 18:05         ` Eric Dumazet
2016-04-18 16:44       ` Butler, Peter
2016-04-15 22:37   ` Rick Jones
2016-04-18 11:27     ` Butler, Peter
2016-04-18 18:22       ` Rick Jones
2016-04-19 14:54         ` Butler, Peter
2016-04-19 15:13           ` Josh Hunt
2016-04-19 15:47             ` Butler, Peter
2016-04-19 15:58           ` Oliver Hartkopp
2016-04-19 16:00             ` Butler, Peter
2016-04-19 16:13             ` David Miller
2016-04-18 12:02     ` Butler, Peter

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).