Re:Re: schedule delay 15ms

[JerryJun <jerryjun123@163.com>]

[-- Attachment #1: Type: text/plain, Size: 3940 bytes --]


Which configurations do you think could cause this delay? During this process, the hardware remained unchanged. We used the following configurations and attempted to disable certain features in the kernel boot parameters, but it didn't work: xenomai.smi=disabled intel_idle.max_cstate=0 intel_pstate=disable nox2apic

At 2025-11-19 00:00:12, "Jan Kiszka" <jan.kiszka@siemens.com> wrote:
>On 18.11.25 16:31, JerryJun wrote:
>> 
>> Strangely enough, this delay wasn't present on the 4.19.89-ipip kernel - it only emerged after upgrading to 5.10.199-dovetail, despite no changes being made to the power management policy or thermal throttling settings during the update. Could this be related to potential instability in the HPET or TSC clock sources? And does upgrading to a new kernel version also necessitate a BIOS update?
>> 
>
>Generally spoken, a newer kernel could always have changed something
>regarding how power tuning is done. Simple scenario: a kernel config
>switch was renaming or gained a new dependency that is not fulfill ->
>tuning feature is silently lost.
>
>Another reason for different behavior could be the enabling (maybe even
>by default) of a new feature that has unexpected side effects. You may
>check if there are new drivers active, new devices available.
>
>Jan
>
>> At 2025-11-14 13:38:36, "Jan Kiszka" <jan.kiszka@siemens.com> wrote:
>>> On 14.11.25 03:48, JerryJun wrote:
>>>>
>>>>   We added some trace event information, but from this gap point (6208.695948-> 6208.708726), it appears that the Cobalt kernel was frozen and no interrupt handling occurred.
>>>>    include trace point events :
>>>>     irq_vectors:call_function_entry                    
>>>>     irq_vectors:call_function_exit                     
>>>>     irq_vectors:call_function_single_entry             
>>>>     irq_vectors:call_function_single_exit   
>>>>
>>>
>>> If there are no interrupt in that phase while, e.g., some timer
>>> interrupt should have occurred, then you should definitely check for
>>> hardware/firmware related delays that the OS cannot influence. Or for
>>> power-saving / thermal-throttling causes.
>>>
>>> Jan
>>>
>>>> 在 2025-11-13 13:31:54，"Jan Kiszka" <jan.kiszka@siemens.com> 写道：
>>>>> On 13.11.25 05:00, JerryJun wrote:
>>>>>>
>>>>>> We conducted tests on the kernel version v5.10.245-cip65-dovetail1. The test results are attached. During this period, we also performed latency tests, which showed a 10ms delay at 7174.313414 -> 7174.323086.  irq 114 is net driver interrupt 。
>>>>>>
>>>>>
>>>>>         swapper     0 [003]  7174.313414:                     irq:irq_pipeline_entry: irq=114
>>>>>         swapper     0 [003]  7174.323086:                     irq:irq_pipeline_entry: irq=33027
>>>>>
>>>>> So, that is the gap. Did you also turn on irq_pipeline_exit? I'm not 
>>>>> seeing that event in the trace. Furthermore, I always for get about 
>>>>> that, it is useful to have irq_vectors events because those include 
>>>>> timer irqs. But it is unlikely that we missed any in that gap because 
>>>>> they tend to trigger activities like cobalt_core:cobalt_timer_expire.
>>>>>
>>>>> What is more likely here is that your hardware or firmware is not 
>>>>> letting IRQs through. We could confirm that if the issue is reproducible 
>>>>> while have function tracing enabled. That comes with high overhead, but 
>>>>> the detailed result can tell us what the kernel did while there were no 
>>>>> interrupts.
>>>>>
>>>>> There is also native tooling by the kernel to detect "hardware noise",
>>>>> see https://docs.kernel.org/tools/rtla/rtla-hwnoise.html. Due to its 
>>>>> nature, this noise would affect Xenomai equally.
>>>>>
>>>>> Jan
>>>>>
>>>>> -- 
>>>>> Siemens AG, Foundational Technologies
>>>>> Linux Expert Center
>>>
>>>
>>> -- 
>>> Siemens AG, Foundational Technologies
>>> Linux Expert Center
>
>-- 
>Siemens AG, Foundational Technologies
>Linux Expert Center

[-- Attachment #2: 5.10.199-dovetail.txt --]
[-- Type: text/plain, Size: 8154 bytes --]

CONFIG_ACPI_ADXL=y
CONFIG_APPLE_PROPERTIES=y
CONFIG_AQTION=m
CONFIG_ARCH_CLOCKSOURCE_INIT=y
CONFIG_ARCH_CPUIDLE_HALTPOLL=y
CONFIG_ARCH_HAS_COPY_MC=y
CONFIG_ARCH_HAS_CPU_FINALIZE_INIT=y
CONFIG_ARCH_HAS_DEBUG_VM_PGTABLE=y
CONFIG_ARCH_HAS_DEBUG_WX=y
CONFIG_ARCH_HAS_GIGANTIC_PAGE=y
CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE=y
CONFIG_ARCH_HAS_PTE_DEVMAP=y
CONFIG_ARCH_HAS_SET_DIRECT_MAP=y
CONFIG_ARCH_STACKWALK=y
CONFIG_ARCH_USE_MEMREMAP_PROT=y
CONFIG_ARCH_USE_SYM_ANNOTATIONS=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM=y
CONFIG_ARCH_WANT_LD_ORPHAN_WARN=y
CONFIG_AS_AVX512=y
CONFIG_AS_IS_GNU=y
CONFIG_ASM_MODVERSIONS=y
CONFIG_AS_SHA1_NI=y
CONFIG_AS_SHA256_NI=y
CONFIG_AS_TPAUSE=y
CONFIG_AS_VERSION=23502
CONFIG_ATA_FORCE=y
CONFIG_BLK_CGROUP_RWSTAT=y
CONFIG_BLK_DEV_INTEGRITY_T10=m
CONFIG_BLK_PM=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
CONFIG_BUILDTIME_TABLE_SORT=y
CONFIG_CC_CAN_LINK_STATIC=y
CONFIG_CC_CAN_LINK=y
CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_CC_HAS_INT128=y
CONFIG_CC_HAS_KASAN_GENERIC=y
CONFIG_CC_HAS_RETURN_THUNK=y
CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
CONFIG_CC_VERSION_TEXT="gcc (Debian 10.2.1-6) 10.2.1 20210110"
CONFIG_CGROUP_BPF=y
CONFIG_CHELSIO_INLINE_CRYPTO=y
CONFIG_COMEDI_NI_ROUTING=m
CONFIG_CPU_IBPB_ENTRY=y
CONFIG_CPU_IBRS_ENTRY=y
CONFIG_CPU_SUP_HYGON=y
CONFIG_CPU_SUP_ZHAOXIN=y
CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=m
CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305=m
CONFIG_CRYPTO_BLAKE2B=m
CONFIG_CRYPTO_ECC=m
CONFIG_CRYPTO_ESSIV=m
CONFIG_CRYPTO_LIB_AES=y
CONFIG_CRYPTO_LIB_ARC4=m
CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC=y
CONFIG_CRYPTO_LIB_CHACHA_GENERIC=m
CONFIG_CRYPTO_LIB_DES=m
CONFIG_CRYPTO_LIB_POLY1305_GENERIC=m
CONFIG_CRYPTO_LIB_POLY1305_RSIZE=11
CONFIG_CRYPTO_LIB_SHA256=y
CONFIG_CRYPTO_SKCIPHER2=y
CONFIG_CRYPTO_SKCIPHER=m
CONFIG_CRYPTO_USER_API_ENABLE_OBSOLETE=y
CONFIG_CRYPTO_XXHASH=m
CONFIG_CRYPTO_ZSTD=m
CONFIG_DEBUG_FS_ALLOW_ALL=y
CONFIG_DEBUG_MISC=y
CONFIG_DECOMPRESS_ZSTD=y
CONFIG_DEFAULT_INIT=""
CONFIG_DIMLIB=y
CONFIG_DMA_OPS=y
CONFIG_DOVETAIL_LEGACY_SYSCALL_RANGE=y
CONFIG_DOVETAIL=y
CONFIG_DRM_AMD_DC_DCN=y
CONFIG_DRM_GEM_SHMEM_HELPER=y
CONFIG_DRM_I915_FENCE_TIMEOUT=10000
CONFIG_DRM_I915_FORCE_PROBE=""
CONFIG_DRM_I915_HEARTBEAT_INTERVAL=2500
CONFIG_DRM_I915_MAX_REQUEST_BUSYWAIT=8000
CONFIG_DRM_I915_PREEMPT_TIMEOUT=640
CONFIG_DRM_I915_STOP_TIMEOUT=100
CONFIG_DRM_I915_TIMESLICE_DURATION=1
CONFIG_DRM_I915_USERFAULT_AUTOSUSPEND=250
CONFIG_DRM_TTM_DMA_PAGE_POOL=y
CONFIG_DRM_TTM_HELPER=m
CONFIG_DRM_VRAM_HELPER=m
CONFIG_DYNAMIC_EVENTS=y
CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y
CONFIG_EFI_DEV_PATH_PARSER=y
CONFIG_EFI_EARLYCON=y
CONFIG_EFI_GENERIC_STUB_INITRD_CMDLINE_LOADER=y
CONFIG_EFI_STUB=y
CONFIG_ETHTOOL_NETLINK=y
CONFIG_FB_BACKLIGHT=m
CONFIG_FW_CACHE=y
CONFIG_FW_LOADER_PAGED_BUF=y
CONFIG_GCC_VERSION=100201
CONFIG_GENERIC_ENTRY=y
CONFIG_GENERIC_GETTIMEOFDAY=y
CONFIG_GENERIC_PTDUMP=y
CONFIG_GENERIC_VDSO_TIME_NS=y
CONFIG_GPIO_CDEV_V1=y
CONFIG_GPIO_CDEV=y
CONFIG_HARDIRQS_SW_RESEND=y
CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
CONFIG_HAVE_ARCH_KCSAN=y
CONFIG_HAVE_ARCH_SECCOMP=y
CONFIG_HAVE_ARCH_STACKLEAK=y
CONFIG_HAVE_ARCH_USERFAULTFD_WP=y
CONFIG_HAVE_ASM_MODVERSIONS=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_DOVETAIL=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_HAVE_EISA=y
CONFIG_HAVE_FAST_GUP=y
CONFIG_HAVE_FUNCTION_ARG_ACCESS_API=y
CONFIG_HAVE_GENERIC_VDSO=y
CONFIG_HAVE_IRQ_PIPELINE=y
CONFIG_HAVE_KERNEL_ZSTD=y
CONFIG_HAVE_KVM_NO_POLL=y
CONFIG_HAVE_MOVE_PMD=y
CONFIG_HAVE_PCI=y
CONFIG_HAVE_PERCPU_PREEMPT_COUNT=y
CONFIG_HAVE_POSIX_CPU_TIMERS_TASK_WORK=y
CONFIG_HAVE_STACK_VALIDATION=y
CONFIG_HAVE_STATIC_CALL_INLINE=y
CONFIG_HAVE_STATIC_CALL=y
CONFIG_HIBERNATION_SNAPSHOT_DEV=y
CONFIG_HMM_MIRROR=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_IA32_FEAT_CTL=y
CONFIG_IAVF=m
CONFIG_IMA_MEASURE_ASYMMETRIC_KEYS=y
CONFIG_IMA_QUEUE_EARLY_BOOT_KEYS=y
CONFIG_INET_TABLE_PERTURB_ORDER=16
CONFIG_INFINIBAND_VIRT_DMA=y
CONFIG_INIT_STACK_NONE=y
CONFIG_INTEL_RAPL_CORE=m
CONFIG_IOASID=y
CONFIG_IOMMU_DMA=y
CONFIG_IO_URING=y
CONFIG_IO_WQ=y
CONFIG_IP6_NF_MATCH_SRH=m
CONFIG_IPMI_PLAT_DATA=y
CONFIG_IPVLAN_L3S=y
CONFIG_IRQ_MSI_IOMMU=y
CONFIG_IRQ_PIPELINE=y
CONFIG_ISDN_CAPI=y
CONFIG_JOYSTICK_IFORCE_232=m
CONFIG_JOYSTICK_IFORCE_USB=m
CONFIG_KCMP=y
CONFIG_KVM_GUEST=y
CONFIG_KVM_WERROR=y
CONFIG_KVM_XFER_TO_GUEST_WORK=y
CONFIG_LD_ORPHAN_WARN=y
CONFIG_LD_VERSION=235020000
CONFIG_LEDS_TRIGGER_AUDIO=m
CONFIG_LIB_MEMNEQ=y
CONFIG_LLD_VERSION=0
CONFIG_LOCALVERSION="-dovetail-xenomai-3.2.4-20240228"
CONFIG_LSM="lockdown,yama,loadpin,safesetid,integrity,bpf"
CONFIG_MAGIC_SYSRQ_SERIAL_SEQUENCE=""
CONFIG_MAPPING_DIRTY_HELPERS=y
CONFIG_MDIO_DEVRES=m
CONFIG_MEDIA_CONTROLLER_DVB=y
CONFIG_MEDIA_CONTROLLER_REQUEST_API=y
CONFIG_MEDIA_PLATFORM_SUPPORT=y
CONFIG_MEDIA_TEST_SUPPORT=y
CONFIG_MEMREGION=y
CONFIG_MMC_SDHCI_IO_ACCESSORS=y
CONFIG_MMU_GATHER_RCU_TABLE_FREE=y
CONFIG_MMU_GATHER_TABLE_FREE=y
CONFIG_MXM_WMI=m
CONFIG_NET_DEVLINK=y
CONFIG_NETFILTER_XT_TARGET_MASQUERADE=m
CONFIG_NET_REDIRECT=y
CONFIG_NET_VENDOR_GOOGLE=y
CONFIG_NET_VENDOR_PENSANDO=y
CONFIG_NET_VENDOR_XILINX=y
CONFIG_NF_CONNTRACK_BRIDGE=m
CONFIG_NF_CT_PROTO_GRE=y
CONFIG_NF_DUP_NETDEV=m
CONFIG_NF_FLOW_TABLE_INET=m
CONFIG_NF_FLOW_TABLE_IPV4=m
CONFIG_NF_FLOW_TABLE_IPV6=m
CONFIG_NF_FLOW_TABLE=m
CONFIG_NF_LOG_NETDEV=m
CONFIG_NF_NAT_MASQUERADE=y
CONFIG_NFS_DISABLE_UDP_SUPPORT=y
CONFIG_NF_TABLES_ARP=y
CONFIG_NF_TABLES_BRIDGE=m
CONFIG_NF_TABLES_INET=y
CONFIG_NF_TABLES_IPV4=y
CONFIG_NF_TABLES_IPV6=y
CONFIG_NF_TABLES_NETDEV=y
CONFIG_NFT_CONNLIMIT=m
CONFIG_NFT_DUP_IPV4=m
CONFIG_NFT_DUP_IPV6=m
CONFIG_NFT_DUP_NETDEV=m
CONFIG_NFT_FIB_IPV4=m
CONFIG_NFT_FIB_IPV6=m
CONFIG_NFT_FIB=m
CONFIG_NFT_FWD_NETDEV=m
CONFIG_NFT_OBJREF=m
CONFIG_NFT_OSF=m
CONFIG_NFT_REJECT_INET=m
CONFIG_NFT_REJECT_IPV4=m
CONFIG_NFT_REJECT_IPV6=m
CONFIG_NFT_SOCKET=m
CONFIG_NFT_SYNPROXY=m
CONFIG_NFT_TPROXY=m
CONFIG_NFT_TUNNEL=m
CONFIG_NFT_XFRM=m
CONFIG_NVDIMM_KEYS=y
CONFIG_NVMEM_SYSFS=y
CONFIG_PAGE_REPORTING=y
CONFIG_PARAVIRT_CLOCK=y
CONFIG_PARAVIRT=y
CONFIG_PATA_TIMINGS=y
CONFIG_PCIE_BUS_DEFAULT=y
CONFIG_POSIX_CPU_TIMERS_TASK_WORK=y
CONFIG_POWER_SUPPLY_HWMON=y
CONFIG_PREEMPTION=y
CONFIG_PROC_PID_ARCH_STATUS=y
CONFIG_PROC_THERMAL_MMIO_RAPL=y
CONFIG_PTP_1588_CLOCK_KVM=y
CONFIG_RAID6_PQ_BENCHMARK=y
CONFIG_RANDOM_TRUST_BOOTLOADER=y
CONFIG_RD_ZSTD=y
CONFIG_REGMAP_I2C=m
CONFIG_REGMAP_SPI=m
CONFIG_SATA_HOST=y
CONFIG_SECURITY_SELINUX_SID2STR_CACHE_SIZE=256
CONFIG_SECURITY_SELINUX_SIDTAB_HASH_BITS=9
CONFIG_SERIAL_8250_DWLIB=y
CONFIG_SERIAL_MCTRL_GPIO=y
CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
CONFIG_SKB_EXTENSIONS=y
CONFIG_SND_HDA_GENERIC_LEDS=y
CONFIG_SND_INTEL_DSP_CONFIG=m
CONFIG_SND_INTEL_NHLT=y
CONFIG_SND_SOC_INTEL_APL=m
CONFIG_SND_SOC_INTEL_CFL=m
CONFIG_SND_SOC_INTEL_CNL=m
CONFIG_SND_SOC_INTEL_GLK=m
CONFIG_SND_SOC_INTEL_KBL=m
CONFIG_SND_SOC_INTEL_SKL=m
CONFIG_SND_SOC_INTEL_SKYLAKE_COMMON=m
CONFIG_SND_SOC_INTEL_SKYLAKE_FAMILY=m
CONFIG_SND_USB_AUDIO_USE_MEDIA_CONTROLLER=y
CONFIG_SPECULATION_MITIGATIONS=y
CONFIG_SPI_DYNAMIC=y
CONFIG_SYMBOLIC_ERRNAME=y
CONFIG_TASKS_RCU_GENERIC=y
CONFIG_TASKS_TRACE_RCU=y
CONFIG_TIME_NS=y
CONFIG_TIPC_CRYPTO=y
CONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=y
CONFIG_TREE_RCU=y
CONFIG_TTY_PRINTK_LEVEL=6
CONFIG_UBIFS_FS_ZSTD=y
CONFIG_UNIX_SCM=y
CONFIG_USB_AUTOSUSPEND_DELAY=2
CONFIG_USB_COMMON=m
CONFIG_VHOST_IOTLB=m
CONFIG_VHOST_MENU=y
CONFIG_VIDEO_V4L2_I2C=y
CONFIG_VIRTIO_DMA_SHARED_BUFFER=m
CONFIG_VMAP_PFN=y
CONFIG_VSOCKETS_LOOPBACK=m
CONFIG_WATCHDOG_OPEN_TIMEOUT=0
CONFIG_WLAN_VENDOR_MICROCHIP=y
CONFIG_X86_HV_CALLBACK_VECTOR=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=12
CONFIG_X86_IOPL_IOPERM=y
CONFIG_X86_UMIP=y
CONFIG_X86_VMX_FEATURE_NAMES=y
CONFIG_X86_VSMP=y
CONFIG_X86_X2APIC=y
CONFIG_XENO_REVISION_LEVEL=4
CONFIG_XENO_VERSION_MINOR=2
CONFIG_XENO_VERSION_STRING="3.2.4"
CONFIG_XFRM_AH=m
CONFIG_XFRM_ESP=m
CONFIG_XFS_SUPPORT_V4=y
CONFIG_XXHASH=y
CONFIG_ZSTD_DECOMPRESS=y

[-- Attachment #3: 4.19.89-ipipe.txt --]
[-- Type: text/plain, Size: 5464 bytes --]

CONFIG_ACPI_PROCESSOR_CSTATE=y
CONFIG_ANON_INODES=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
CONFIG_ARCH_HAS_REFCOUNT=y
CONFIG_ARCH_HAS_SG_CHAIN=y
CONFIG_ARCH_HAS_UACCESS_MCSAFE=y
CONFIG_ARCH_HAS_ZONE_DEVICE=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARM_GIC_MAX_NR=1
CONFIG_AT803X_PHY=m
CONFIG_AUDIT_TREE=y
CONFIG_AUDIT_WATCH=y
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_BALLOON_COMPACTION=y
CONFIG_BCH=m
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_BLK_DEV_SX8=m
CONFIG_BT_HCIUART_MRVL=y
CONFIG_BUILDTIME_EXTABLE_SORT=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CAPI_AVM=y
CONFIG_CAPI_EICON=y
CONFIG_CC_HAS_STACKPROTECTOR_NONE=y
CONFIG_CHASH=m
CONFIG_CHR_DEV_OSST=m
CONFIG_CIFS_ACL=y
CONFIG_COMPACTION=y
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_DECNET=m
CONFIG_DECNET_NF_GRABULATOR=m
CONFIG_DEFAULT_CFQ=y
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_DEFAULT_SECURITY=""
CONFIG_DMA_DIRECT_OPS=y
CONFIG_DMA_VIRT_OPS=y
CONFIG_DOUBLEFAULT=y
CONFIG_DRM_AMD_DC_DCN1_0=y
CONFIG_DSCC4=m
CONFIG_DSCC4_PCI_RST=y
CONFIG_DSCC4_PCISYNC=y
CONFIG_EXOFS_FS=m
CONFIG_EXT4_ENCRYPTION=y
CONFIG_EXT4_FS_ENCRYPTION=y
CONFIG_F2FS_FS_ENCRYPTION=y
CONFIG_FB_BACKLIGHT=y
CONFIG_FS_ENCRYPTION=m
CONFIG_GCC_VERSION=80300
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GIGASET_BASE=m
CONFIG_GIGASET_CAPI=y
CONFIG_GIGASET_M101=m
CONFIG_GIGASET_M105=m
CONFIG_HAVE_COPY_THREAD_TLS=y
CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
CONFIG_HAVE_GENERIC_GUP=y
CONFIG_HAVE_IPIPE_SUPPORT=y
CONFIG_HAVE_IPIPE_TRACER_SUPPORT=y
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_HP100=m
CONFIG_HSA_AMD=m
CONFIG_HYSDN_CAPI=y
CONFIG_HYSDN=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INFINIBAND_CXGB3=m
CONFIG_INFINIBAND_NES=m
CONFIG_INTEL_MIC_BUS=m
CONFIG_INTEL_MIC_HOST=m
CONFIG_INTEL_MIC_X100_DMA=m
CONFIG_INTEL_PMC_IPC=m
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IOMMU_HELPER=y
CONFIG_IOSCHED_CFQ=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IPIPE_CORE_APIREV=2
CONFIG_IPIPE_CORE=y
CONFIG_IPIPE_HAVE_HOSTRT=y
CONFIG_IPIPE_TARGET_APIREV=2
CONFIG_IPIPE_WANT_APIREV_2=y
CONFIG_IPIPE_WANT_PTE_PINNING=y
CONFIG_IPIPE=y
CONFIG_ISDN_CAPI_CAPI20=m
CONFIG_ISDN_CAPI=m
CONFIG_ISDN_DIVAS_BRIPCI=y
CONFIG_ISDN_DIVAS_DIVACAPI=m
CONFIG_ISDN_DIVAS=m
CONFIG_ISDN_DIVAS_MAINT=m
CONFIG_ISDN_DIVAS_PRIPCI=y
CONFIG_ISDN_DIVAS_USERIDI=m
CONFIG_ISDN_DRV_AVMB1_AVM_CS=m
CONFIG_ISDN_DRV_AVMB1_B1PCI=m
CONFIG_ISDN_DRV_AVMB1_B1PCIV4=y
CONFIG_ISDN_DRV_AVMB1_B1PCMCIA=m
CONFIG_ISDN_DRV_AVMB1_C4=m
CONFIG_ISDN_DRV_AVMB1_T1PCI=m
CONFIG_ISDN_DRV_GIGASET=m
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_KEYS_COMPAT=y
CONFIG_LEDS_LT3593=m
CONFIG_LOCALVERSION="-ipipe-9-xenomai-3.1-20201027"
CONFIG_MAC80211_RC_MINSTREL_HT=y
CONFIG_MAY_USE_DEVLINK=m
CONFIG_MIC_COSM=m
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_MIGRATION=y
CONFIG_MTD_M25P80=m
CONFIG_MTD_NAND_BCH=m
CONFIG_MTD_NAND_CAFE=m
CONFIG_MTD_NAND_DISKONCHIP=m
CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADDRESS=0
CONFIG_MTD_NAND_ECC_BCH=y
CONFIG_MTD_NAND_ECC=m
CONFIG_MTD_NAND=m
CONFIG_MTD_NAND_NANDSIM=m
CONFIG_MTD_NAND_RICOH=m
CONFIG_MTD_SM_COMMON=m
CONFIG_MXM_WMI=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_DEVLINK=m
CONFIG_NET_VENDOR_HP=y
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_NAT_IPV4=m
CONFIG_NF_NAT_IPV6=m
CONFIG_NF_NAT_MASQUERADE_IPV4=y
CONFIG_NF_NAT_MASQUERADE_IPV6=y
CONFIG_NF_NAT_NEEDED=y
CONFIG_NF_NAT_PROTO_DCCP=y
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_SCTP=y
CONFIG_NF_NAT_PROTO_UDPLITE=y
CONFIG_NO_BOOTMEM=y
CONFIG_NODES_SPAN_OTHER_NODES=y
CONFIG_NUMA_BALANCING=y
CONFIG_OPTIMIZE_INLINING=y
CONFIG_ORE=m
CONFIG_PLUGIN_HOSTCC=""
CONFIG_QUOTACTL_COMPAT=y
CONFIG_REED_SOLOMON_DEC16=y
CONFIG_REGMAP_I2C=y
CONFIG_REGMAP_SPI=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_SCIF_BUS=m
CONFIG_SCIF=m
CONFIG_SCSI_LPFC=m
CONFIG_SCSI_OSD_DPRINT_SENSE=1
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=1
CONFIG_SENSORS_ADS1015=m
CONFIG_SERIAL_8250_MOXA=m
CONFIG_SGI_IOC4=m
CONFIG_SND_SST_IPC_ACPI=m
CONFIG_SND_SST_IPC=m
CONFIG_THUNDERBOLT=m
CONFIG_TRACING_EVENTS_GPIO=y
CONFIG_USB_COMMON=y
CONFIG_USB_HWA_HCD=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_WHCI_HCD=m
CONFIG_USB_WUSB_CBAF=m
CONFIG_USB_WUSB=m
CONFIG_UWB_HWA=m
CONFIG_UWB_I1480U=m
CONFIG_UWB=m
CONFIG_UWB_WHCI=m
CONFIG_VHOST_RING=m
CONFIG_VIDEO_USBVISION=m
CONFIG_VOP_BUS=m
CONFIG_VOP=m
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_X86_INTEL_UMIP=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_XENO_OPT_HOSTRT=y
CONFIG_XENO_OPT_STATS_IRQS=y
CONFIG_XENO_REVISION_LEVEL=0
CONFIG_XENO_VERSION_MINOR=1
CONFIG_XENO_VERSION_STRING="3.1"
CONFIG_XXHASH=m
CONFIG_ZSTD_DECOMPRESS=m