Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[V JobNickname]

When I porting my ARM platform to Kernel 3.10.0 the network throughput
of TCP tx is about 160 Mbps.

After I applied patch-3.10.37, the TCP tx drop to 8x Mbps.
I found that after I disabled CONFIG_HIGH_RES_TIMERS and NO_HZ_IDEL,
the throughput can back to 16x Mbps.

I try to find the more step patches and tile patch-3.10.17, the
throughput with enabled NO_HZ_IDEL or HIGH_RES_TIMERS can still have
16x Mbps
After applied patch-3.10.17-18, it start to drop the performance.

More detail trace, the first commit of 3.10.18, tcp: TSO packets
automatic sizing, cause to this issue.
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/diff/?id=5e25ba5003ee5de0ba2be56bfd54d16d4b1b028d

I don't know why this TSO patch affect my network since the network
driver doesn't enable NETIF_F_TSO.
But I
(dev->hw_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM;
dev->features |= dev->hw_features;)

After look into the commit difference,
I can just doubt if function "tcp_update_pacing_rate" can work well
with NO_HZ_IDEL or HIGH_RES_TIMERS, since it base on jiffie to
calculate the rate instead of human nenosecond of new timer
arch(clocksource/clockevent)

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[Eric Dumazet]

On Wed, 2014-05-07 at 00:17 +0800, V JobNickname wrote:
> When I porting my ARM platform to Kernel 3.10.0 the network throughput
> of TCP tx is about 160 Mbps.
> 
> After I applied patch-3.10.37, the TCP tx drop to 8x Mbps.
> I found that after I disabled CONFIG_HIGH_RES_TIMERS and NO_HZ_IDEL,
> the throughput can back to 16x Mbps.
> 
> I try to find the more step patches and tile patch-3.10.17, the
> throughput with enabled NO_HZ_IDEL or HIGH_RES_TIMERS can still have
> 16x Mbps
> After applied patch-3.10.17-18, it start to drop the performance.
> 
> More detail trace, the first commit of 3.10.18, tcp: TSO packets
> automatic sizing, cause to this issue.
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/diff/?id=5e25ba5003ee5de0ba2be56bfd54d16d4b1b028d
> 
> I don't know why this TSO patch affect my network since the network
> driver doesn't enable NETIF_F_TSO.
> But I
> (dev->hw_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM;
> dev->features |= dev->hw_features;)
> 
> After look into the commit difference,
> I can just doubt if function "tcp_update_pacing_rate" can work well
> with NO_HZ_IDEL or HIGH_RES_TIMERS, since it base on jiffie to
> calculate the rate instead of human nenosecond of new timer
> arch(clocksource/clockevent)
> --

I cant see any effect for this patch explaining a 50% reduction in
speed, unless a special qdisc setup maybe.

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[Eric Dumazet]

On Tue, 2014-05-06 at 09:30 -0700, Eric Dumazet wrote:

> I cant see any effect for this patch explaining a 50% reduction in
> speed, unless a special qdisc setup maybe.

Oh wait, a buggy driver could explain the problem.

Some drivers really expected TX completion to happen as a side effect of
subsequent transmits.

You'll have to give more details on the configuration you are using.

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[V JobNickname]

2014-05-07 0:41 GMT+08:00 Eric Dumazet <eric.dumazet@gmail.com>:
> On Tue, 2014-05-06 at 09:30 -0700, Eric Dumazet wrote:
>
>> I cant see any effect for this patch explaining a 50% reduction in
>> speed, unless a special qdisc setup maybe.
>
> Oh wait, a buggy driver could explain the problem.
>
> Some drivers really expected TX completion to happen as a side effect of
> subsequent transmits.
>
> You'll have to give more details on the configuration you are using.
>
>
>
Does the "buggy driver" here mean the NIC driver or some higher level
network stack driver?
The xmit function of the used NIC driver is to stuff DMA channel, is
there have "expected Tx completion" behavior in the driver's flow ?

The more detail:

1. The platform's NIC is compilable with dwmac, use driver
ethernet/stmicro/stmmac
     some feature controls in plat_stmmacenet_data are set as
        .phy_bus_name = "stmmac",
        .bus_id = 0,
        .phy_addr = -1, /*auto phy by pooling by mdio bus */
        .has_gmac = 1,
        .enh_desc = 0,
        .tx_coe = 1,
        .rx_coe = STMMAC_RX_COE_TYPE2,

2. with the ./ethtool -k eth0
        Features for eth0:
        rx-checksumming: on
        tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
        scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
        tcp-segmentation-offload: off
        tx-tcp-segmentation: off [fixed]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
        udp-fragmentation-offload: off [fixed]
        generic-segmentation-offload: on
        generic-receive-offload: on
        large-receive-offload: off [fixed]
        rx-vlan-offload: off [fixed]
        tx-vlan-offload: off [fixed]
        ntuple-filters: off [fixed]
        receive-hashing: off [fixed]
        highdma: on [fixed]
        rx-vlan-filter: off [fixed]
        vlan-challenged: off [fixed]
        tx-lockless: off [fixed]
        netns-local: off [fixed]
        tx-gso-robust: off [fixed]
        tx-fcoe-segmentation: off [fixed]
        tx-gre-segmentation: off [fixed]
        tx-udp_tnl-segmentation: off [fixed]
        fcoe-mtu: off [fixed]
        tx-nocache-copy: on
        loopback: off [fixed]
        rx-fcs: off [fixed]
        rx-all: off [fixed]
        tx-vlan-stag-hw-insert: off [fixed]
        rx-vlan-stag-hw-parse: off [fixed]
        rx-vlan-stag-filter: off [fixed]
3.
The following is my configuration file .config
#
# Automatically generated file; DO NOT EDIT.
# Linux/arm 3.10.17 Kernel Configuration
#
CONFIG_ARM=y
CONFIG_SYS_SUPPORTS_APM_
EMULATION=y
CONFIG_HAVE_PROC_CPU=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_VECTORS_BASE=0xffff0000
CONFIG_ARM_PATCH_PHYS_VIRT=y
CONFIG_GENERIC_BUG=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y

#
# General setup
#
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE="arm-linux-"
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
CONFIG_FHANDLE=y
# CONFIG_AUDIT is not set
CONFIG_HAVE_GENERIC_HARDIRQS=y

#
# IRQ subsystem
#
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_HARDIRQS_SW_RESEND=y
CONFIG_KTIME_SCALAR=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y

#
# CPU/Task time and stats accounting
#
CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_IRQ_TIME_ACCOUNTING is not set
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set

#
# RCU Subsystem
#
# CONFIG_TREE_PREEMPT_RCU is not set
CONFIG_TINY_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU=y
# CONFIG_RCU_STALL_COMMON is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_RCU_BOOST is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=14
# CONFIG_CGROUPS is not set
# CONFIG_CHECKPOINT_RESTORE is not set
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
CONFIG_IPC_NS=y
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
CONFIG_UIDGID_CONVERTED=y
# CONFIG_UIDGID_STRICT_TYPE_CHECKS is not set
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
# CONFIG_RELAY is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
# CONFIG_RD_BZIP2 is not set
# CONFIG_RD_LZMA is not set
# CONFIG_RD_XZ is not set
# CONFIG_RD_LZO is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_HAVE_UID16=y
CONFIG_HOTPLUG=y
CONFIG_EXPERT=y
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_EMBEDDED=y
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_USE_VMALLOC=y

#
# Kernel Performance Events And Counters
#
# CONFIG_PERF_EVENTS is not set
# CONFIG_VM_EVENT_COUNTERS is not set
CONFIG_COMPAT_BRK=y
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_JUMP_LABEL=y
# CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_GENERIC_IDLE_POLL_SETUP=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_REL=y
CONFIG_CLONE_BACKWARDS=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_OLD_SIGACTION=y

#
# GCOV-based kernel profiling
#
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_MODULE_SIG is not set
CONFIG_BLOCK=y
# CONFIG_LBDAF is not set
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_BSGLIB is not set
# CONFIG_BLK_DEV_INTEGRITY is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
CONFIG_DEFAULT_NOOP=y
CONFIG_DEFAULT_IOSCHED="noop"
CONFIG_UNINLINE_SPIN_UNLOCK=y
# CONFIG_FREEZER is not set

#
# System Type
#
CONFIG_MMU=y
# CONFIG_ARCH_MULTIPLATFORM is not set
# CONFIG_ARCH_INTEGRATOR is not set
# CONFIG_ARCH_REALVIEW is not set
# CONFIG_ARCH_VERSATILE is not set
# CONFIG_ARCH_AT91 is not set
# CONFIG_ARCH_CLPS711X is not set
# CONFIG_ARCH_GEMINI is not set
# CONFIG_ARCH_EBSA110 is not set
# CONFIG_ARCH_EP93XX is not set
# CONFIG_ARCH_FOOTBRIDGE is not set
# CONFIG_ARCH_NETX is not set
# CONFIG_ARCH_IOP13XX is not set
# CONFIG_ARCH_IOP32X is not set
# CONFIG_ARCH_IOP33X is not set
# CONFIG_ARCH_IXP4XX is not set
# CONFIG_ARCH_DOVE is not set
# CONFIG_ARCH_KIRKWOOD is not set
# CONFIG_ARCH_MV78XX0 is not set
# CONFIG_ARCH_ORION5X is not set
# CONFIG_ARCH_MMP is not set
# CONFIG_ARCH_KS8695 is not set
# CONFIG_ARCH_W90X900 is not set
# CONFIG_ARCH_LPC32XX is not set
# CONFIG_ARCH_PXA is not set
# CONFIG_ARCH_MSM is not set
# CONFIG_ARCH_SHMOBILE is not set
# CONFIG_ARCH_RPC is not set
# CONFIG_ARCH_SA1100 is not set
# CONFIG_ARCH_S3C24XX is not set
# CONFIG_ARCH_S3C64XX is not set
# CONFIG_ARCH_S5P64X0 is not set
# CONFIG_ARCH_S5PC100 is not set
# CONFIG_ARCH_S5PV210 is not set
# CONFIG_ARCH_EXYNOS is not set
# CONFIG_ARCH_SHARK is not set
# CONFIG_ARCH_U300 is not set
# CONFIG_ARCH_DAVINCI is not set
# CONFIG_ARCH_OMAP1 is not set
CONFIG_ARCH_MYSOC=y
# CONFIG_PLAT_SPEAR is not set

#
# Processor Type
#
CONFIG_CPU_ARM926T=y
CONFIG_CPU_32v5=y
CONFIG_CPU_ABRT_EV5TJ=y
CONFIG_CPU_PABRT_LEGACY=y
CONFIG_CPU_CACHE_VIVT=y
CONFIG_CPU_COPY_V4WB=y
CONFIG_CPU_TLB_V4WBI=y
CONFIG_CPU_CP15=y
CONFIG_CPU_CP15_MMU=y
CONFIG_CPU_USE_DOMAINS=y

#
# Processor Features
#
# CONFIG_ARCH_PHYS_ADDR_T_64BIT is not set
# CONFIG_ARM_THUMB is not set
# CONFIG_CPU_ICACHE_DISABLE is not set
# CONFIG_CPU_DCACHE_DISABLE is not set
# CONFIG_CPU_DCACHE_WRITETHROUGH is not set
# CONFIG_CPU_CACHE_ROUND_ROBIN is not set
CONFIG_NEED_KUSER_HELPERS=y
CONFIG_KUSER_HELPERS=y
# CONFIG_CACHE_L2X0 is not set
CONFIG_ARM_L1_CACHE_SHIFT=5
CONFIG_ARM_NR_BANKS=8

#
# Bus support
#
# CONFIG_PCI_SYSCALL is not set
# CONFIG_PCCARD is not set

#
# Kernel Features
#
CONFIG_VMSPLIT_3G=y
# CONFIG_VMSPLIT_2G is not set
# CONFIG_VMSPLIT_1G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_ARCH_NR_GPIO=0
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_AEABI=y
# CONFIG_OABI_COMPAT is not set
# CONFIG_ARCH_SPARSEMEM_DEFAULT is not set
# CONFIG_ARCH_SELECT_MEMORY_MODEL is not set
CONFIG_HAVE_ARCH_PFN_VALID=y
# CONFIG_HIGHMEM is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_HAVE_MEMBLOCK=y
# CONFIG_HAVE_BOOTMEM_INFO_NODE is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=999999
# CONFIG_COMPACTION is not set
# CONFIG_PHYS_ADDR_T_64BIT is not set
CONFIG_ZONE_DMA_FLAG=0
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
# CONFIG_CROSS_MEMORY_ATTACH is not set
CONFIG_NEED_PER_CPU_KM=y
# CONFIG_CLEANCACHE is not set
CONFIG_FORCE_MAX_ZONEORDER=11
CONFIG_ALIGNMENT_TRAP=y
# CONFIG_UACCESS_WITH_MEMCPY is not set
# CONFIG_SECCOMP is not set
# CONFIG_CC_STACKPROTECTOR is not set

#
# Boot options
#
# CONFIG_USE_OF is not set
CONFIG_ATAGS=y
# CONFIG_DEPRECATED_PARAM_STRUCT is not set
CONFIG_ZBOOT_ROM_TEXT=0x0
CONFIG_ZBOOT_ROM_BSS=0x0
CONFIG_CMDLINE="root=/ console=1 mem=64M"
CONFIG_CMDLINE_FROM_BOOTLOADER=y
# CONFIG_CMDLINE_EXTEND is not set
# CONFIG_CMDLINE_FORCE is not set
# CONFIG_XIP_KERNEL is not set
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
# CONFIG_AUTO_ZRELADDR is not set

#
# CPU Power Management
#
# CONFIG_CPU_IDLE is not set
# CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set

#
# Floating point emulation
#

#
# At least one emulation must be selected
#
# CONFIG_VFP is not set

#
# Userspace binary formats
#
CONFIG_BINFMT_ELF=y
CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y
CONFIG_BINFMT_SCRIPT=y
# CONFIG_HAVE_AOUT is not set
# CONFIG_BINFMT_MISC is not set
# CONFIG_COREDUMP is not set

#
# Power management options
#
# CONFIG_SUSPEND is not set
# CONFIG_PM_RUNTIME is not set
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ARM_CPU_SUSPEND is not set
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_DIAG is not set
CONFIG_UNIX=y
# CONFIG_UNIX_DIAG is not set
CONFIG_XFRM=y
# CONFIG_XFRM_USER is not set
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_PNP=y
# CONFIG_IP_PNP_DHCP is not set
# CONFIG_IP_PNP_BOOTP is not set
# CONFIG_IP_PNP_RARP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE_DEMUX is not set
# CONFIG_NET_IP_TUNNEL is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_NET_IPVTI is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
CONFIG_INET_XFRM_MODE_TRANSPORT=y
CONFIG_INET_XFRM_MODE_TUNNEL=y
CONFIG_INET_XFRM_MODE_BEET=y
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_INET_UDP_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
# CONFIG_NETFILTER is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
# CONFIG_BRIDGE is not set
CONFIG_HAVE_NET_DSA=y
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set
# CONFIG_BATMAN_ADV is not set
# CONFIG_OPENVSWITCH is not set
# CONFIG_VSOCKETS is not set
# CONFIG_NETLINK_MMAP is not set
# CONFIG_NETLINK_DIAG is not set
CONFIG_BQL=y
# CONFIG_BPF_JIT is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
CONFIG_WIRELESS=y
# CONFIG_CFG80211 is not set
# CONFIG_LIB80211 is not set

#
# CFG80211 needs to be enabled for MAC80211
#
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
# CONFIG_CEPH_LIB is not set
# CONFIG_NFC is not set
CONFIG_HAVE_BPF_JIT=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_FW_LOADER_USER_HELPER=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_GENERIC_CPU_DEVICES is not set
# CONFIG_DMA_SHARED_BUFFER is not set
# CONFIG_CMA is not set

#
# Bus devices
#
# CONFIG_CONNECTOR is not set
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_COW_COMMON is not set
# CONFIG_BLK_DEV_LOOP is not set
# CONFIG_BLK_DEV_DRBD is not set
# CONFIG_BLK_DEV_NBD is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=2
CONFIG_BLK_DEV_RAM_SIZE=8192
# CONFIG_BLK_DEV_XIP is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_BLK_DEV_RBD is not set

#
# Misc devices
#
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_ATMEL_SSC is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_SRAM is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_93CX6 is not set

#
# Texas Instruments shared transport line discipline
#

#
# Altera FPGA firmware download module
#

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
# CONFIG_RAID_ATTRS is not set
# CONFIG_SCSI is not set
# CONFIG_SCSI_DMA is not set
# CONFIG_SCSI_NETLINK is not set
# CONFIG_ATA is not set
# CONFIG_MD is not set
CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
# CONFIG_BONDING is not set
# CONFIG_DUMMY is not set
# CONFIG_EQUALIZER is not set
CONFIG_MII=y
# CONFIG_NET_TEAM is not set
# CONFIG_MACVLAN is not set
# CONFIG_VXLAN is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_TUN is not set
# CONFIG_VETH is not set

#
# CAIF transport drivers
#

#
# Distributed Switch Architecture drivers
#
# CONFIG_NET_DSA_MV88E6XXX is not set
# CONFIG_NET_DSA_MV88E6060 is not set
# CONFIG_NET_DSA_MV88E6XXX_NEED_PPU is not set
# CONFIG_NET_DSA_MV88E6131 is not set
# CONFIG_NET_DSA_MV88E6123_61_65 is not set
CONFIG_ETHERNET=y
# CONFIG_NET_CADENCE is not set
# CONFIG_NET_VENDOR_BROADCOM is not set
# CONFIG_NET_CALXEDA_XGMAC is not set
# CONFIG_NET_VENDOR_CIRRUS is not set
# CONFIG_DM9000 is not set
# CONFIG_DNET is not set
# CONFIG_NET_VENDOR_FARADAY is not set
# CONFIG_NET_VENDOR_INTEL is not set
# CONFIG_NET_VENDOR_MARVELL is not set
# CONFIG_NET_VENDOR_MICREL is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
# CONFIG_ETHOC is not set
# CONFIG_NET_VENDOR_SEEQ is not set
# CONFIG_NET_VENDOR_SMSC is not set
CONFIG_NET_VENDOR_STMICRO=y
CONFIG_STMMAC_ETH=y
CONFIG_STMMAC_PLATFORM=y
# CONFIG_STMMAC_DA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
CONFIG_PHYLIB=y

#
# MII PHY device drivers
#
# CONFIG_AT803X_PHY is not set
# CONFIG_AMD_PHY is not set
# CONFIG_MARVELL_PHY is not set
# CONFIG_DAVICOM_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_CICADA_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_BROADCOM_PHY is not set
# CONFIG_BCM87XX_PHY is not set
# CONFIG_ICPLUS_PHY is not set
CONFIG_REALTEK_PHY=y
# CONFIG_NATIONAL_PHY is not set
# CONFIG_STE10XP is not set
# CONFIG_LSI_ET1011C_PHY is not set
# CONFIG_MICREL_PHY is not set
# CONFIG_FIXED_PHY is not set
# CONFIG_MDIO_BITBANG is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
CONFIG_WLAN=y
# CONFIG_HOSTAP is not set
# CONFIG_WL_TI is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
# CONFIG_WAN is not set
# CONFIG_ISDN is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set
# CONFIG_INPUT_SPARSEKMAP is not set
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
# CONFIG_INPUT_MOUSEDEV is not set
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
# CONFIG_INPUT_KEYBOARD is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set

#
# Hardware I/O ports
#
# CONFIG_SERIO is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_N_GSM is not set
# CONFIG_TRACE_SINK is not set
CONFIG_DEVKMEM=y

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_DEPRECATED_OPTIONS=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set
# CONFIG_SERIAL_8250_DW is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_ARC is not set
# CONFIG_TTY_PRINTK is not set
# CONFIG_HVC_DCC is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_R3964 is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_I2C is not set
# CONFIG_SPI is not set

#
# Qualcomm MSM SSBI bus support
#
# CONFIG_SSBI is not set
# CONFIG_HSI is not set

#
# PPS support
#
CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set
# CONFIG_NTP_PPS is not set

#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
# CONFIG_PPS_CLIENT_LDISC is not set
# CONFIG_PPS_CLIENT_GPIO is not set

#
# PPS generators support
#

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=y

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# CONFIG_PTP_1588_CLOCK_PCH is not set
CONFIG_ARCH_HAVE_CUSTOM_GPIO_H=y
CONFIG_GPIO_DEVRES=y
# CONFIG_W1 is not set
# CONFIG_POWER_SUPPLY is not set
# CONFIG_POWER_AVS is not set
# CONFIG_HWMON is not set
# CONFIG_THERMAL is not set
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y

#
# Broadcom specific AMBA
#
# CONFIG_BCMA is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_CROS_EC is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_REGULATOR is not set
# CONFIG_MEDIA_SUPPORT is not set

#
# Graphics support
#
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
# CONFIG_FB is not set
# CONFIG_EXYNOS_VIDEO is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Console display driver support
#
CONFIG_DUMMY_CONSOLE=y
# CONFIG_SOUND is not set

#
# HID support
#
# CONFIG_HID is not set
# CONFIG_USB_ARCH_HAS_OHCI is not set
# CONFIG_USB_ARCH_HAS_EHCI is not set
# CONFIG_USB_ARCH_HAS_XHCI is not set
# CONFIG_USB_SUPPORT is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_EDAC is not set
CONFIG_RTC_LIB=y
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_VIRT_DRIVERS is not set

#
# Virtio drivers
#
# CONFIG_VIRTIO_MMIO is not set

#
# Microsoft Hyper-V guest support
#
# CONFIG_STAGING is not set

#
# Hardware Spinlock drivers
#
# CONFIG_MAILBOX is not set
# CONFIG_IOMMU_SUPPORT is not set

#
# Remoteproc drivers
#
# CONFIG_STE_MODEM_RPROC is not set

#
# Rpmsg drivers
#
# CONFIG_PM_DEVFREQ is not set
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
# CONFIG_PWM is not set
# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set

#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
# CONFIG_EXT3_FS is not set
# CONFIG_EXT4_FS is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
# CONFIG_FS_POSIX_ACL is not set
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
# CONFIG_QUOTA is not set
# CONFIG_QUOTACTL is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
# CONFIG_MSDOS_FS is not set
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
# CONFIG_TMPFS is not set
# CONFIG_HUGETLB_PAGE is not set
# CONFIG_CONFIGFS_FS is not set
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_LOGFS is not set
# CONFIG_CRAMFS is not set
CONFIG_SQUASHFS=y
# CONFIG_SQUASHFS_XATTR is not set
CONFIG_SQUASHFS_ZLIB=y
# CONFIG_SQUASHFS_LZO is not set
CONFIG_SQUASHFS_XZ=y
# CONFIG_SQUASHFS_4K_DEVBLK_SIZE is not set
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX6FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_PSTORE is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_F2FS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_SWAP is not set
CONFIG_ROOT_NFS=y
# CONFIG_NFSD is not set
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
# CONFIG_SUNRPC_DEBUG is not set
# CONFIG_CEPH_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=y
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_MAC_ROMAN is not set
# CONFIG_NLS_MAC_CELTIC is not set
# CONFIG_NLS_MAC_CENTEURO is not set
# CONFIG_NLS_MAC_CROATIAN is not set
# CONFIG_NLS_MAC_CYRILLIC is not set
# CONFIG_NLS_MAC_GAELIC is not set
# CONFIG_NLS_MAC_GREEK is not set
# CONFIG_NLS_MAC_ICELAND is not set
# CONFIG_NLS_MAC_INUIT is not set
# CONFIG_NLS_MAC_ROMANIAN is not set
# CONFIG_NLS_MAC_TURKISH is not set
# CONFIG_NLS_UTF8 is not set

#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not set
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=0
# CONFIG_MAGIC_SYSRQ is not set
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_READABLE_ASM is not set
# CONFIG_UNUSED_SYMBOLS is not set
# CONFIG_DEBUG_FS is not set
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_SECTION_MISMATCH is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_LOCKUP_DETECTOR is not set
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
# CONFIG_DETECT_HUNG_TASK is not set
# CONFIG_SCHED_DEBUG is not set
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_BUGVERBOSE is not set
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_WRITECOUNT is not set
# CONFIG_DEBUG_MEMORY_INIT is not set
# CONFIG_DEBUG_LIST is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
# CONFIG_BOOT_PRINTK_DELAY is not set

#
# RCU Debugging
#
# CONFIG_PROVE_RCU_DELAY is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_TRACE is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_NOTIFIER_ERROR_INJECTION is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
# CONFIG_FTRACE is not set
# CONFIG_RBTREE_TEST is not set
# CONFIG_INTERVAL_TREE_TEST is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
# CONFIG_TEST_STRING_HELPERS is not set
# CONFIG_TEST_KSTRTOX is not set
# CONFIG_STRICT_DEVMEM is not set
CONFIG_ARM_UNWIND=y
CONFIG_DEBUG_USER=y
CONFIG_DEBUG_LL=y
CONFIG_DEBUG_MYSOC_UART=y
# CONFIG_DEBUG_LL_UART_NONE is not set
# CONFIG_DEBUG_ICEDCC is not set
# CONFIG_DEBUG_SEMIHOSTING is not set
CONFIG_DEBUG_MYSOC_UART_PORT=1
CONFIG_DEBUG_LL_INCLUDE="mach/debug-macro.S"
CONFIG_UNCOMPRESS_INCLUDE="mach/uncompress.h"
CONFIG_EARLY_PRINTK=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
# CONFIG_CRYPTO_MANAGER is not set
# CONFIG_CRYPTO_MANAGER2 is not set
# CONFIG_CRYPTO_USER is not set
# CONFIG_CRYPTO_GF128MUL is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
# CONFIG_CRYPTO_CBC is not set
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
# CONFIG_CRYPTO_ECB is not set
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
# CONFIG_CRYPTO_CMAC is not set
# CONFIG_CRYPTO_HMAC is not set
# CONFIG_CRYPTO_XCBC is not set
# CONFIG_CRYPTO_VMAC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32 is not set
# CONFIG_CRYPTO_GHASH is not set
# CONFIG_CRYPTO_MD4 is not set
# CONFIG_CRYPTO_MD5 is not set
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
# CONFIG_CRYPTO_SHA1 is not set
# CONFIG_CRYPTO_SHA1_ARM is not set
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_ARM is not set
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set

#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_ZLIB is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
# CONFIG_CRYPTO_HW is not set
# CONFIG_BINARY_PRINTF is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IO=y
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not set
# CONFIG_CRC_T10DIF is not set
# CONFIG_CRC_ITU_T is not set
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
# CONFIG_CRC8 is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_XZ_DEC=y
# CONFIG_XZ_DEC_X86 is not set
# CONFIG_XZ_DEC_POWERPC is not set
# CONFIG_XZ_DEC_IA64 is not set
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
# CONFIG_XZ_DEC_SPARC is not set
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_GENERIC_ATOMIC64=y
CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y
# CONFIG_AVERAGE is not set
# CONFIG_CORDIC is not set
# CONFIG_DDR is not set
# CONFIG_VIRTUALIZATION is not set

RE: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[David Laight]

From: V JobNickname
> 2014-05-07 0:41 GMT+08:00 Eric Dumazet <eric.dumazet@gmail.com>:
> > On Tue, 2014-05-06 at 09:30 -0700, Eric Dumazet wrote:
> >
> >> I cant see any effect for this patch explaining a 50% reduction in
> >> speed, unless a special qdisc setup maybe.
> >
> > Oh wait, a buggy driver could explain the problem.
> >
> > Some drivers really expected TX completion to happen as a side effect of
> > subsequent transmits.
> >
> > You'll have to give more details on the configuration you are using.

> Does the "buggy driver" here mean the NIC driver or some higher level
> network stack driver?
> The xmit function of the used NIC driver is to stuff DMA channel, is
> there have "expected Tx completion" behavior in the driver's flow ?

Some of the TCP transmit flow control requires that the NIC driver
free the skb as soon as it is transmitted (if not before) and also
to limit the number of bytes of data queued for transmit.

This is done to give fair(er) access to the network when there are
some connections doing file transfer with large window and others
doing command-response.

	David

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[Eric Dumazet]

On Wed, 2014-05-07 at 22:27 +0800, V JobNickname wrote:
> 2014-05-07 0:41 GMT+08:00 Eric Dumazet <eric.dumazet@gmail.com>:
> > On Tue, 2014-05-06 at 09:30 -0700, Eric Dumazet wrote:
> >
> >> I cant see any effect for this patch explaining a 50% reduction in
> >> speed, unless a special qdisc setup maybe.
> >
> > Oh wait, a buggy driver could explain the problem.
> >
> > Some drivers really expected TX completion to happen as a side effect of
> > subsequent transmits.
> >
> > You'll have to give more details on the configuration you are using.
> >
> >
> >
> Does the "buggy driver" here mean the NIC driver or some higher level
> network stack driver?
> The xmit function of the used NIC driver is to stuff DMA channel, is
> there have "expected Tx completion" behavior in the driver's flow ?
> 
> The more detail:
> 
> 1. The platform's NIC is compilable with dwmac, use driver
> ethernet/stmicro/stmmac
>      some feature controls in plat_stmmacenet_data are set as
>         .phy_bus_name = "stmmac",
>         .bus_id = 0,
>         .phy_addr = -1, /*auto phy by pooling by mdio bus */
>         .has_gmac = 1,
>         .enh_desc = 0,
>         .tx_coe = 1,
>         .rx_coe = STMMAC_RX_COE_TYPE2,

Right, this driver seems to implement a 'coalescing tx strategy' using a
40ms timer.

Try to lower STMMAC_COAL_TX_TIMER ?

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[V JobNickname]

Try to lower
#define STMMAC_COAL_TX_TIMER    10000
or
#define STMMAC_TX_FRAMES    0 (this will casue to skip coalescing path)
can't solve the issue.

2014-05-07 23:45 GMT+08:00 Eric Dumazet <eric.dumazet@gmail.com>:
> On Wed, 2014-05-07 at 22:27 +0800, V JobNickname wrote:
>> 2014-05-07 0:41 GMT+08:00 Eric Dumazet <eric.dumazet@gmail.com>:
>> > On Tue, 2014-05-06 at 09:30 -0700, Eric Dumazet wrote:
>> >
>> >> I cant see any effect for this patch explaining a 50% reduction in
>> >> speed, unless a special qdisc setup maybe.
>> >
>> > Oh wait, a buggy driver could explain the problem.
>> >
>> > Some drivers really expected TX completion to happen as a side effect of
>> > subsequent transmits.
>> >
>> > You'll have to give more details on the configuration you are using.
>> >
>> >
>> >
>> Does the "buggy driver" here mean the NIC driver or some higher level
>> network stack driver?
>> The xmit function of the used NIC driver is to stuff DMA channel, is
>> there have "expected Tx completion" behavior in the driver's flow ?
>>
>> The more detail:
>>
>> 1. The platform's NIC is compilable with dwmac, use driver
>> ethernet/stmicro/stmmac
>>      some feature controls in plat_stmmacenet_data are set as
>>         .phy_bus_name = "stmmac",
>>         .bus_id = 0,
>>         .phy_addr = -1, /*auto phy by pooling by mdio bus */
>>         .has_gmac = 1,
>>         .enh_desc = 0,
>>         .tx_coe = 1,
>>         .rx_coe = STMMAC_RX_COE_TYPE2,
>
> Right, this driver seems to implement a 'coalescing tx strategy' using a
> 40ms timer.
>
> Try to lower STMMAC_COAL_TX_TIMER ?
>
>
>

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[Neal Cardwell]

On Tue, May 6, 2014 at 12:17 PM, V JobNickname <workofv@gmail.com> wrote:
> More detail trace, the first commit of 3.10.18, tcp: TSO packets
> automatic sizing, cause to this issue.
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/diff/?id=5e25ba5003ee5de0ba2be56bfd54d16d4b1b028d

To provide another data point, it would be interesting to know what
performance you see with v3.15-rc4. Linux 3.15 has the following
patch:

 740b0f1841f6  tcp: switch rtt estimations to usec resolution

That patch switches tcp_update_pacing_rate() to use microsecond RTTs
for its calculation. That may help things in your case. (As Eric says,
more details would help.)

> I don't know why this TSO patch affect my network since the network
> driver doesn't enable NETIF_F_TSO.

Keep in mind that anytime the Linux TCP code or commit descriptions
talk about "TSO", the same applies to GSO. Is GSO enabled for your
device?

neal

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[V JobNickname]

2014-05-07 0:48 GMT+08:00 Neal Cardwell <ncardwell@google.com>:
> On Tue, May 6, 2014 at 12:17 PM, V JobNickname <workofv@gmail.com> wrote:
>> More detail trace, the first commit of 3.10.18, tcp: TSO packets
>> automatic sizing, cause to this issue.
>> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/diff/?id=5e25ba5003ee5de0ba2be56bfd54d16d4b1b028d
>
> To provide another data point, it would be interesting to know what
> performance you see with v3.15-rc4. Linux 3.15 has the following
> patch:
>
>  740b0f1841f6  tcp: switch rtt estimations to usec resolution
>
> That patch switches tcp_update_pacing_rate() to use microsecond RTTs
> for its calculation. That may help things in your case. (As Eric says,
> more details would help.)
>
>> I don't know why this TSO patch affect my network since the network
>> driver doesn't enable NETIF_F_TSO.
>
> Keep in mind that anytime the Linux TCP code or commit descriptions
> talk about "TSO", the same applies to GSO. Is GSO enabled for your
> device?
>
> neal

Porting to the v3.15-rc4 from main stream, the performance is well when
 enabled NO_HZ_IDEL or HIGH_RES_TIMERS.

I also try the version before commit 740b0f1841f6  tcp: switch rtt
estimations to usec resolution, i.e
363ec392352 net: add skb_mstamp infrastructure,
also not cause to performance drop. So I think my case is regardless
to use jiffie or nesc in tcp_update_pacing_rate().

Then I try more eraly version and found that in 3.14-rc2 thes case was
hit but solved at 3.14-rc3.
(I use "git checkout -b 3.14-rc2 v3.14-rc2" and "git checkout -b
3.14-rc3 v3.14-rc3"  to get taged version,
and use "git checkout 3.14-rc2/3.14-rc3 to switch between". Hope my
usage are correct, since I'm not familer with git.)

I think some commit between 3.14-rc2 and 3.14-rc3 can help find the
root cause of my case.

ps.
But I don't know how to use git to update from rc2 to rc3 by each
commit(or maybe binary search commit log), and find that one commit.

For example, I would like to following logs start from 3.14-rc2 tag in
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/log/?h=linux-3.14.y&ofs=1650

Can I,

git checkout 3.14-rc2 -- switch to 3.14-rc2
git checout 07b0e5b10258b48e5edfb6c8ac156f05510eb775 (ASoC: da9055:
Fix device registration of PMIC and CODEC devices)
will the commit between this commit and 3.14-rc2 also updated?
if this commit solve my case, I can try some one commit between it and 3.14-rc2.
such as
git checkout a49f56eec54d864ba0fda838e4c8bf5c72f3eb08 (microblaze: Fix
a typo when disabling stack protection)
should I revert to 3.14-rc2 first, or the git will done it?

Thanks

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[Eric Dumazet]

On Wed, 2014-05-07 at 23:51 +0800, V JobNickname wrote:

> 
> Porting to the v3.15-rc4 from main stream, the performance is well when
>  enabled NO_HZ_IDEL or HIGH_RES_TIMERS.
> 
> I also try the version before commit 740b0f1841f6  tcp: switch rtt
> estimations to usec resolution, i.e
> 363ec392352 net: add skb_mstamp infrastructure,
> also not cause to performance drop. So I think my case is regardless
> to use jiffie or nesc in tcp_update_pacing_rate().
> 
> Then I try more eraly version and found that in 3.14-rc2 thes case was
> hit but solved at 3.14-rc3.
> (I use "git checkout -b 3.14-rc2 v3.14-rc2" and "git checkout -b
> 3.14-rc3 v3.14-rc3"  to get taged version,
> and use "git checkout 3.14-rc2/3.14-rc3 to switch between". Hope my
> usage are correct, since I'm not familer with git.)
> 
> I think some commit between 3.14-rc2 and 3.14-rc3 can help find the
> root cause of my case.
> 
> ps.
> But I don't know how to use git to update from rc2 to rc3 by each
> commit(or maybe binary search commit log), and find that one commit.
> 
> For example, I would like to following logs start from 3.14-rc2 tag in
> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/log/?h=linux-3.14.y&ofs=1650
> 
> Can I,
> 
> git checkout 3.14-rc2 -- switch to 3.14-rc2
> git checout 07b0e5b10258b48e5edfb6c8ac156f05510eb775 (ASoC: da9055:
> Fix device registration of PMIC and CODEC devices)
> will the commit between this commit and 3.14-rc2 also updated?
> if this commit solve my case, I can try some one commit between it and 3.14-rc2.
> such as
> git checkout a49f56eec54d864ba0fda838e4c8bf5c72f3eb08 (microblaze: Fix
> a typo when disabling stack protection)
> should I revert to 3.14-rc2 first, or the git will done it?
> 
> Thanks

I am wondering if you are asking for git bisect maybe ?

https://www.kernel.org/pub/software/scm/git/docs/git-bisect.html

But in this case I am not sure it will help.

An old bug can sit for years, and finally be discovered when some other
change is done.

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[Eric Dumazet]

On Wed, 2014-05-07 at 09:25 -0700, Eric Dumazet wrote:

> I am wondering if you are asking for git bisect maybe ?
> 
> https://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
> 
> But in this case I am not sure it will help.
> 
> An old bug can sit for years, and finally be discovered when some other
> change is done.


Yes, following code is rather crappy :

        if (priv->tx_coal_frames > priv->tx_count_frames) {
                priv->hw->desc->clear_tx_ic(desc);
                priv->xstats.tx_reset_ic_bit++;
                mod_timer(&priv->txtimer,
                          STMMAC_COAL_TIMER(priv->tx_coal_timer));
        } else
                priv->tx_count_frames = 0;

40 ms is already a big timer, and restarting it every time another frame
is queued is a total disaster.

Re: Is 3.18 patch "The tcp: TSO packets automatic sizing" will cause to Troughput drop when enable NO_HZ_IDEL or HIGH_RES_TIMERS ?

[V JobNickname]

2014-05-08 0:25 GMT+08:00 Eric Dumazet <eric.dumazet@gmail.com>:
> On Wed, 2014-05-07 at 23:51 +0800, V JobNickname wrote:
>
>>
>> Porting to the v3.15-rc4 from main stream, the performance is well when
>>  enabled NO_HZ_IDEL or HIGH_RES_TIMERS.
>>
>> I also try the version before commit 740b0f1841f6  tcp: switch rtt
>> estimations to usec resolution, i.e
>> 363ec392352 net: add skb_mstamp infrastructure,
>> also not cause to performance drop. So I think my case is regardless
>> to use jiffie or nesc in tcp_update_pacing_rate().
>>
>> Then I try more eraly version and found that in 3.14-rc2 thes case was
>> hit but solved at 3.14-rc3.
>> (I use "git checkout -b 3.14-rc2 v3.14-rc2" and "git checkout -b
>> 3.14-rc3 v3.14-rc3"  to get taged version,
>> and use "git checkout 3.14-rc2/3.14-rc3 to switch between". Hope my
>> usage are correct, since I'm not familer with git.)
>>
>> I think some commit between 3.14-rc2 and 3.14-rc3 can help find the
>> root cause of my case.
>>
>> ps.
>> But I don't know how to use git to update from rc2 to rc3 by each
>> commit(or maybe binary search commit log), and find that one commit.
>>
>> For example, I would like to following logs start from 3.14-rc2 tag in
>> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/log/?h=linux-3.14.y&ofs=1650
>>
>> Can I,
>>
>> git checkout 3.14-rc2 -- switch to 3.14-rc2
>> git checout 07b0e5b10258b48e5edfb6c8ac156f05510eb775 (ASoC: da9055:
>> Fix device registration of PMIC and CODEC devices)
>> will the commit between this commit and 3.14-rc2 also updated?
>> if this commit solve my case, I can try some one commit between it and 3.14-rc2.
>> such as
>> git checkout a49f56eec54d864ba0fda838e4c8bf5c72f3eb08 (microblaze: Fix
>> a typo when disabling stack protection)
>> should I revert to 3.14-rc2 first, or the git will done it?
>>
>> Thanks
>
> I am wondering if you are asking for git bisect maybe ?
>
> https://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
>
> But in this case I am not sure it will help.
>
> An old bug can sit for years, and finally be discovered when some other
> change is done.
>
>
I still try the git bisect.
Note that, in my case the "good" means "8xMbps" and bad is "17x Mbps",
I start at 3.17-rc2 as good to find first bad commit.
The results are

git bisect start
# good: [b28a960c42fcd9cfc987441fa6d1c1a471f0f9ed] Linux 3.14-rc2
git bisect good b28a960c42fcd9cfc987441fa6d1c1a471f0f9ed
# bad: [6d0abeca3242a88cab8232e4acd7e2bf088f3bc2] Linux 3.14-rc3
git bisect bad 6d0abeca3242a88cab8232e4acd7e2bf088f3bc2
# bad: [cb6ef42e516cb8948f15e4b70dc03af8020050a2] EDAC: Correct
workqueue setup path
git bisect bad cb6ef42e516cb8948f15e4b70dc03af8020050a2
# bad: [960b589f86c74ce582922fcb996103271081f4de] bridge: Properly
check if local fdb entry can be deleted in br_fdb_change_mac_address
git bisect bad 960b589f86c74ce582922fcb996103271081f4de
# bad: [f41f03196041f91acad2b6d2b3e1f800aed60100] Merge branch
'master' of git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf
git bisect bad f41f03196041f91acad2b6d2b3e1f800aed60100
# good: [028b86b7671d17eb9a614924eb46d5efb9db931a] Merge branch
'fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jesse/openvswitch
git bisect good 028b86b7671d17eb9a614924eb46d5efb9db931a
# good: [4a5ab4e224288403b0b4b6b8c4d339323150c312] tcp: remove 1ms
offset in srtt computation
git bisect good 4a5ab4e224288403b0b4b6b8c4d339323150c312
# good: [51292c0735eb2d9e29115cbf6264845e19a6c77d] netfilter: nft_ct:
fix missing NFT_CT_L3PROTOCOL key in validity checks
git bisect good 51292c0735eb2d9e29115cbf6264845e19a6c77d
# good: [0165d9325d6a3cf856e2cbbe64a0f4635ac75893] netfilter:
nf_tables: fix racy rule deletion
git bisect good 0165d9325d6a3cf856e2cbbe64a0f4635ac75893
# good: [2fb91ddbf8e1dcb207e1e2085473aeaeff975102] netfilter:
nft_rbtree: fix data handling of end interval elements
git bisect good 2fb91ddbf8e1dcb207e1e2085473aeaeff975102
# good: [6d8c00d58e9e484fdc41aaaf62e5d8364efe375a] netfilter:
nf_tables: unininline nft_trace_packet()
git bisect good 6d8c00d58e9e484fdc41aaaf62e5d8364efe375a

f41f03196041f91acad2b6d2b3e1f800aed60100 is the first bad commit
(Merge branch 'master' of
git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf)

But this merge commit only touch netfilter related code, which I
doesn't enabled in .config.
Another problem is the 3.14-rc2 tag is at 2014-02-10, but some of
above search is early than the day, and not between 3.14-rc2 to
3.14-rc3 when I check with "git log"
So, this is not help for my case...
Thanks