* n_tty_write() going into schedule but NOT coming out
@ 2013-03-30 12:35 Vineet Gupta
2013-04-01 13:57 ` Vineet Gupta
2013-04-03 5:44 ` Ilya Zykov
0 siblings, 2 replies; 10+ messages in thread
From: Vineet Gupta @ 2013-03-30 12:35 UTC (permalink / raw)
To: lkml, linux-serial; +Cc: Ingo Molnar, Peter Zijlstra, Jiri Slaby, Peter Hurley
[-- Attachment #1: Type: text/plain, Size: 1501 bytes --]
Hi,
I've been stress testing ARC Linux 3.8 (same happens for 3.9-rc3 as well). The
setup has 3 telnet sessions, each running find . -name "*" in a loop.
The platform is a FPGA @ 80 MHz, running a single core ARC700 so kernel .config
has !SMP and PREEMPT_NONE.
After ~10 mins of run, I see that one of the telnet session gets stuck (and later
the 2nd one as well), while system is still alive, 3rd telnet is running find merrily.
[ARCLinux]$ ps
....
7 root 0:00 inetd
62 root 0:00 -/bin/sh
64 root 1:34 telnetd -i -l /bin/sh
65 root 0:00 /bin/sh
75 root 1:47 telnetd -i -l /bin/sh
76 root 0:00 /bin/sh
79 root 0:53 telnetd -i -l /bin/sh
80 root 0:00 /bin/sh
281 root 0:00 find / -name * <--- stuck
358 root 0:03 find / -name * <--- stuck
377 root 0:00 find / -name *
378 root 0:00 ps
Hung find task is sitting in the schedule() call in n_tty_write()
[ARCLinux]$ cat /proc/281/stack
[<8065945e>] n_tty_write+0x23a/0x424
[<80655cd4>] tty_write+0x1ac/0x2d4
[<805976ba>] vfs_write+0x92/0x110
[<80597816>] sys_write+0x4e/0x88
[<8050e780>] ret_from_system_call+0x0/0x4
This task never resumes out of schedule() - verified by putting a hardware
breakpoint on next insn - using a JTAG host debugger.
Attached are .config, /proc/281/sched, /proc/schedstat, /proc/sched_debug
My knowledge of schedular is close to none, hence any tips to debug this further
would be much appreciated.
TIA,
-Vineet
[-- Attachment #2: .config --]
[-- Type: text/plain, Size: 21892 bytes --]
#
# Automatically generated file; DO NOT EDIT.
# Linux/arc 3.8.0 Kernel Configuration
#
CONFIG_ARC=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_GENERIC_CSUM=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_MMU=y
CONFIG_NO_IOPORT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_BINFMT_ELF=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
# CONFIG_NO_DMA is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_HAVE_IRQ_WORK=y
#
# General setup
#
# CONFIG_EXPERIMENTAL is not set
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE="arc-elf32-"
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_DEFAULT_HOSTNAME="ARCLinux"
# CONFIG_SWAP is not set
# CONFIG_SYSVIPC is not set
# CONFIG_FHANDLE is not set
# CONFIG_AUDIT is not set
CONFIG_HAVE_GENERIC_HARDIRQS=y
#
# IRQ subsystem
#
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_IRQ_DOMAIN=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
#
# CPU/Task time and stats accounting
#
CONFIG_TICK_CPU_ACCOUNTING=y
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
#
# RCU Subsystem
#
CONFIG_TINY_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
# CONFIG_CGROUPS is not set
# CONFIG_CHECKPOINT_RESTORE is not set
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_NET_NS=y
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
# CONFIG_RELAY is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE="../arc_initramfs_21_tools_upd_abi_v2"
CONFIG_INITRAMFS_ROOT_UID=0
CONFIG_INITRAMFS_ROOT_GID=0
CONFIG_RD_GZIP=y
# CONFIG_RD_BZIP2 is not set
# CONFIG_RD_LZMA is not set
# CONFIG_RD_XZ is not set
# CONFIG_RD_LZO is not set
CONFIG_INITRAMFS_COMPRESSION_NONE=y
# CONFIG_INITRAMFS_COMPRESSION_GZIP is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_EXPERT=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_EMBEDDED=y
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_USE_VMALLOC=y
#
# Kernel Performance Events And Counters
#
# CONFIG_PERF_EVENTS is not set
CONFIG_VM_EVENT_COUNTERS=y
# CONFIG_SLUB_DEBUG is not set
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_GENERIC_SIGALTSTACK=y
CONFIG_CLONE_BACKWARDS=y
#
# GCOV-based kernel profiling
#
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
# CONFIG_MODULE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_MODULE_SIG is not set
CONFIG_BLOCK=y
# CONFIG_LBDAF is not set
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_BSGLIB is not set
# CONFIG_BLK_DEV_INTEGRITY is not set
#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_EFI_PARTITION=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_DEADLINE is not set
# CONFIG_IOSCHED_CFQ is not set
CONFIG_DEFAULT_NOOP=y
CONFIG_DEFAULT_IOSCHED="noop"
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
# CONFIG_FREEZER is not set
#
# ARC Architecture Configuration
#
#
# ARC Platform/SoC/Board
#
CONFIG_ARC_PLAT_FPGA_LEGACY=y
CONFIG_ARC_BOARD_ANGEL4=y
CONFIG_ARC_BOARD_ML509=y
CONFIG_ARC_SERIAL_BAUD=115200
# CONFIG_ARC_HAS_BVCI_LAT_UNIT is not set
CONFIG_ARC_IDE=y
CONFIG_ARC_BLK_DEV_IDEDMA=y
CONFIG_ARCTANGENT_EMAC=y
CONFIG_EMAC_NAPI=y
#
# ARC CPU Configuration
#
# CONFIG_ARC_CPU_750D is not set
CONFIG_ARC_CPU_770=y
# CONFIG_CPU_BIG_ENDIAN is not set
# CONFIG_SMP is not set
CONFIG_ARC_CACHE=y
CONFIG_ARC_CACHE_LINE_SHIFT=6
CONFIG_ARC_HAS_ICACHE=y
CONFIG_ARC_HAS_DCACHE=y
CONFIG_ARC_CACHE_PAGES=y
# CONFIG_ARC_HAS_ICCM is not set
# CONFIG_ARC_HAS_DCCM is not set
CONFIG_ARC_HAS_HW_MPY=y
# CONFIG_ARC_MMU_V1 is not set
# CONFIG_ARC_MMU_V2 is not set
CONFIG_ARC_MMU_V3=y
CONFIG_ARC_PAGE_SIZE_8K=y
# CONFIG_ARC_PAGE_SIZE_16K is not set
# CONFIG_ARC_PAGE_SIZE_4K is not set
# CONFIG_ARC_COMPACT_IRQ_LEVELS is not set
# CONFIG_ARC_FPU_SAVE_RESTORE is not set
CONFIG_ARC_CPU_REL_4_10=y
CONFIG_ARC_HAS_LLSC=y
CONFIG_ARC_HAS_SWAPE=y
# CONFIG_ARC_HAS_RTSC is not set
CONFIG_LINUX_LINK_BASE=0x80000000
CONFIG_ARC_CURR_IN_REG=y
# CONFIG_ARC_MISALIGN_ACCESS is not set
# CONFIG_ARC_STACK_NONEXEC is not set
CONFIG_HZ=100
# CONFIG_ARC_METAWARE_HLINK is not set
CONFIG_ARC_DBG=y
CONFIG_ARC_DW2_UNWIND=y
# CONFIG_ARC_DBG_TLB_PARANOIA is not set
# CONFIG_ARC_DBG_EVENT_TIMELINE is not set
# CONFIG_ARC_DBG_TLB_MISS_COUNT is not set
CONFIG_CMDLINE="print-fatal-signals=1"
# CONFIG_CMDLINE_UBOOT is not set
CONFIG_ARC_BUILTIN_DTB_NAME="angel4"
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_NO_BOOTMEM=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_COMPACTION is not set
# CONFIG_PHYS_ADDR_T_64BIT is not set
CONFIG_ZONE_DMA_FLAG=0
CONFIG_VIRT_TO_BUS=y
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
# CONFIG_CROSS_MEMORY_ATTACH is not set
CONFIG_NEED_PER_CPU_KM=y
# CONFIG_CLEANCACHE is not set
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_DIAG is not set
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=y
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=y
# CONFIG_XFRM_USER is not set
CONFIG_NET_KEY=y
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE_DEMUX is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_NET_IPVTI is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
CONFIG_INET_XFRM_MODE_TRANSPORT=y
CONFIG_INET_XFRM_MODE_TUNNEL=y
CONFIG_INET_XFRM_MODE_BEET=y
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_INET_UDP_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_IPV6 is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETFILTER is not set
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
# CONFIG_BRIDGE is not set
CONFIG_HAVE_NET_DSA=y
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_PHONET is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set
# CONFIG_BATMAN_ADV is not set
# CONFIG_OPENVSWITCH is not set
CONFIG_BQL=y
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_WIRELESS=y
# CONFIG_CFG80211 is not set
# CONFIG_LIB80211 is not set
#
# CFG80211 needs to be enabled for MAC80211
#
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
# CONFIG_NFC is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
# CONFIG_DEVTMPFS_MOUNT is not set
# CONFIG_STANDALONE is not set
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
CONFIG_FW_LOADER=y
# CONFIG_FIRMWARE_IN_KERNEL is not set
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_GENERIC_CPU_DEVICES is not set
# CONFIG_DMA_SHARED_BUFFER is not set
#
# Bus devices
#
# CONFIG_CONNECTOR is not set
# CONFIG_MTD is not set
CONFIG_DTC=y
CONFIG_OF=y
#
# Device Tree and Open Firmware support
#
# CONFIG_PROC_DEVICETREE is not set
# CONFIG_OF_SELFTEST is not set
CONFIG_OF_FLATTREE=y
CONFIG_OF_EARLY_FLATTREE=y
CONFIG_OF_ADDRESS=y
CONFIG_OF_IRQ=y
CONFIG_OF_DEVICE=y
CONFIG_OF_NET=y
# CONFIG_PARPORT is not set
# CONFIG_BLK_DEV is not set
#
# Misc devices
#
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_C2PORT is not set
#
# EEPROM support
#
# CONFIG_EEPROM_93CX6 is not set
#
# Texas Instruments shared transport line discipline
#
#
# Altera FPGA firmware download module
#
CONFIG_HAVE_IDE=y
CONFIG_IDE=y
#
# Please see Documentation/ide/ide.txt for help/info on IDE drives
#
CONFIG_IDE_XFER_MODE=y
# CONFIG_BLK_DEV_IDE_SATA is not set
CONFIG_IDE_GD=y
CONFIG_IDE_GD_ATA=y
# CONFIG_IDE_GD_ATAPI is not set
# CONFIG_BLK_DEV_IDECD is not set
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_IDE_TASK_IOCTL is not set
CONFIG_IDE_PROC_FS=y
#
# IDE chipset support/bugfixes
#
# CONFIG_BLK_DEV_PLATFORM is not set
CONFIG_BLK_DEV_IDEDMA=y
#
# SCSI device support
#
CONFIG_SCSI_MOD=y
# CONFIG_RAID_ATTRS is not set
# CONFIG_SCSI is not set
# CONFIG_SCSI_DMA is not set
# CONFIG_SCSI_NETLINK is not set
# CONFIG_ATA is not set
# CONFIG_MD is not set
CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
# CONFIG_BONDING is not set
# CONFIG_DUMMY is not set
# CONFIG_EQUALIZER is not set
# CONFIG_MII is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_TUN is not set
# CONFIG_VETH is not set
#
# CAIF transport drivers
#
#
# Distributed Switch Architecture drivers
#
# CONFIG_NET_DSA_MV88E6XXX is not set
# CONFIG_NET_DSA_MV88E6060 is not set
# CONFIG_NET_DSA_MV88E6XXX_NEED_PPU is not set
# CONFIG_NET_DSA_MV88E6131 is not set
# CONFIG_NET_DSA_MV88E6123_61_65 is not set
CONFIG_ETHERNET=y
CONFIG_NET_CADENCE=y
# CONFIG_ARM_AT91_ETHER is not set
# CONFIG_MACB is not set
CONFIG_NET_VENDOR_BROADCOM=y
# CONFIG_B44 is not set
# CONFIG_NET_CALXEDA_XGMAC is not set
# CONFIG_DNET is not set
CONFIG_NET_VENDOR_MARVELL=y
# CONFIG_MVMDIO is not set
CONFIG_NET_VENDOR_MICREL=y
# CONFIG_KS8851_MLL is not set
# CONFIG_ETHOC is not set
CONFIG_NET_VENDOR_STMICRO=y
# CONFIG_STMMAC_ETH is not set
CONFIG_NET_VENDOR_WIZNET=y
# CONFIG_WIZNET_W5100 is not set
# CONFIG_WIZNET_W5300 is not set
# CONFIG_PHYLIB is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
CONFIG_WLAN=y
# CONFIG_HOSTAP is not set
# CONFIG_WL_TI is not set
#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
# CONFIG_WAN is not set
# CONFIG_ISDN is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set
# CONFIG_INPUT_SPARSEKMAP is not set
# CONFIG_INPUT_MATRIXKMAP is not set
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
# CONFIG_INPUT_KEYBOARD is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
#
# Hardware I/O ports
#
# CONFIG_SERIO is not set
# CONFIG_GAMEPORT is not set
#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_N_GSM is not set
# CONFIG_TRACE_SINK is not set
# CONFIG_DEVKMEM is not set
#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_XILINX_PS_UART is not set
CONFIG_SERIAL_ARC=y
CONFIG_SERIAL_ARC_CONSOLE=y
CONFIG_SERIAL_ARC_NR_PORTS=1
# CONFIG_TTY_PRINTK is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_RTC is not set
# CONFIG_GEN_RTC is not set
# CONFIG_R3964 is not set
#
# PCMCIA character devices
#
# CONFIG_RAW_DRIVER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_I2C is not set
# CONFIG_SPI is not set
# CONFIG_HSI is not set
#
# PPS support
#
# CONFIG_PPS is not set
#
# PPS generators support
#
#
# PTP clock support
#
# CONFIG_PTP_1588_CLOCK is not set
#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# CONFIG_PTP_1588_CLOCK_PCH is not set
# CONFIG_W1 is not set
# CONFIG_POWER_SUPPLY is not set
# CONFIG_POWER_AVS is not set
# CONFIG_HWMON is not set
# CONFIG_THERMAL is not set
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y
#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y
#
# Broadcom specific AMBA
#
# CONFIG_BCMA is not set
#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_REGULATOR is not set
# CONFIG_MEDIA_SUPPORT is not set
#
# Graphics support
#
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
# CONFIG_FB is not set
# CONFIG_EXYNOS_VIDEO is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set
#
# Console display driver support
#
# CONFIG_VGA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
# CONFIG_SOUND is not set
#
# HID support
#
# CONFIG_HID is not set
# CONFIG_USB_ARCH_HAS_OHCI is not set
# CONFIG_USB_ARCH_HAS_EHCI is not set
# CONFIG_USB_ARCH_HAS_XHCI is not set
# CONFIG_USB_SUPPORT is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
#
# Virtio drivers
#
#
# Microsoft Hyper-V guest support
#
# CONFIG_STAGING is not set
#
# Hardware Spinlock drivers
#
# CONFIG_IOMMU_SUPPORT is not set
#
# Remoteproc drivers (EXPERIMENTAL)
#
#
# Rpmsg drivers (EXPERIMENTAL)
#
# CONFIG_VIRT_DRIVERS is not set
# CONFIG_PM_DEVFREQ is not set
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
# CONFIG_PWM is not set
# CONFIG_IPACK_BUS is not set
#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
# CONFIG_EXT2_FS_POSIX_ACL is not set
# CONFIG_EXT2_FS_SECURITY is not set
# CONFIG_EXT2_FS_XIP is not set
# CONFIG_EXT3_FS is not set
# CONFIG_EXT4_FS is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
# CONFIG_QUOTA is not set
# CONFIG_QUOTACTL is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set
#
# Caches
#
# CONFIG_FSCACHE is not set
#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set
#
# DOS/FAT/NT Filesystems
#
# CONFIG_MSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_TMPFS_XATTR is not set
# CONFIG_HUGETLB_PAGE is not set
# CONFIG_CONFIGFS_FS is not set
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_SWAP is not set
# CONFIG_NFSD is not set
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
# CONFIG_SUNRPC_DEBUG is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_NLS is not set
#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not set
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
# CONFIG_MAGIC_SYSRQ is not set
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_READABLE_ASM is not set
# CONFIG_UNUSED_SYMBOLS is not set
# CONFIG_DEBUG_FS is not set
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_SECTION_MISMATCH is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_LOCKUP_DETECTOR is not set
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
# CONFIG_DETECT_HUNG_TASK is not set
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_WRITECOUNT is not set
# CONFIG_DEBUG_MEMORY_INIT is not set
# CONFIG_DEBUG_LIST is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_TRACE is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_NOTIFIER_ERROR_INJECTION is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_RBTREE_TEST is not set
# CONFIG_INTERVAL_TREE_TEST is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_TEST_KSTRTOX is not set
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_16KSTACKS is not set
#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
# CONFIG_CRYPTO_FIPS is not set
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=m
# CONFIG_CRYPTO_MANAGER is not set
# CONFIG_CRYPTO_MANAGER2 is not set
# CONFIG_CRYPTO_USER is not set
# CONFIG_CRYPTO_GF128MUL is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set
#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set
#
# Block modes
#
# CONFIG_CRYPTO_CBC is not set
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
# CONFIG_CRYPTO_ECB is not set
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set
#
# Hash modes
#
# CONFIG_CRYPTO_HMAC is not set
#
# Digest
#
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_GHASH is not set
# CONFIG_CRYPTO_MD4 is not set
# CONFIG_CRYPTO_MD5 is not set
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
# CONFIG_CRYPTO_SHA1 is not set
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set
#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_ZLIB is not set
# CONFIG_CRYPTO_LZO is not set
#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
CONFIG_CRYPTO_HW=y
# CONFIG_BINARY_PRINTF is not set
#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_IO=y
CONFIG_PERCPU_RWSEM=y
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not set
# CONFIG_CRC_T10DIF is not set
# CONFIG_CRC_ITU_T is not set
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC7 is not set
# CONFIG_LIBCRC32C is not set
# CONFIG_CRC8 is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_DMA=y
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_GENERIC_ATOMIC64=y
CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y
# CONFIG_AVERAGE is not set
# CONFIG_CORDIC is not set
# CONFIG_DDR is not set
[-- Attachment #3: proc_pid_sched.log --]
[-- Type: text/x-log, Size: 2365 bytes --]
find (281, #threads: 1)
---------------------------------------------------------
se.exec_start : 1051320.000000
se.vruntime : 253530.294196
se.sum_exec_runtime : 160.000000
se.statistics.wait_start : 0.000000
se.statistics.sleep_start : 1051320.000000
se.statistics.block_start : 0.000000
se.statistics.sleep_max : 0.000000
se.statistics.block_max : 0.000000
se.statistics.exec_max : 10.000000
se.statistics.slice_max : 10.000000
se.statistics.wait_max : 50.000000
se.statistics.wait_sum : 500.000000
se.statistics.wait_count : 213
se.statistics.iowait_sum : 0.000000
se.statistics.iowait_count : 0
se.nr_migrations : 0
se.statistics.nr_migrations_cold : 0
se.statistics.nr_failed_migrations_affine: 0
se.statistics.nr_failed_migrations_running: 0
se.statistics.nr_failed_migrations_hot: 0
se.statistics.nr_forced_migrations : 0
se.statistics.nr_wakeups : 413
se.statistics.nr_wakeups_sync : 0
se.statistics.nr_wakeups_migrate : 0
se.statistics.nr_wakeups_local : 0
se.statistics.nr_wakeups_remote : 0
se.statistics.nr_wakeups_affine : 0
se.statistics.nr_wakeups_affine_attempts: 0
se.statistics.nr_wakeups_passive : 0
se.statistics.nr_wakeups_idle : 0
avg_atom : 0.751173
avg_per_cpu : 0.000001
nr_switches : 213
nr_voluntary_switches : 1
nr_involuntary_switches : 212
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 0
[-- Attachment #4: proc_sched_debug.log --]
[-- Type: text/x-log, Size: 2577 bytes --]
Sched Debug Version: v0.10, 3.8.0+ #6
ktime : 1777676.065887
sched_clk : 1777670.000000
cpu_clk : 1777670.000000
jiffies : 147767
sysctl_sched
.sysctl_sched_latency : 6.000000
.sysctl_sched_min_granularity : 0.750000
.sysctl_sched_wakeup_granularity : 1.000000
.sysctl_sched_child_runs_first : 0
.sysctl_sched_features : 24187
.sysctl_sched_tunable_scaling : 1 (logaritmic)
cpu#0
.nr_running : 4
.load : 4096
.nr_switches : 4789250
.nr_load_updates : 177762
.nr_uninterruptible : 1
.next_balance : 0.000000
.curr->pid : 435
.clock : 1777670.000000
.cpu_load[0] : 4096
.cpu_load[1] : 3688
.cpu_load[2] : 2971
.cpu_load[3] : 2442
.cpu_load[4] : 2141
.yld_count : 0
.sched_count : 4789368
.sched_goidle : 472
.ttwu_count : 5529500
.ttwu_local : 0
cfs_rq[0]:
.exec_clock : 1738250.000000
.MIN_vruntime : 615584.894138
.min_vruntime : 615579.894138
.max_vruntime : 615587.894138
.spread : 3.000000
.spread0 : 0.000000
.nr_spread_over : 39
.nr_running : 4
.load : 4096
rt_rq[0]:
.rt_nr_running : 0
.rt_throttled : 0
.rt_time : 0.000000
.rt_runtime : 950.000000
runnable tasks:
task PID tree-key switches prio exec-runtime sum-exec sum-sleep
----------------------------------------------------------------------------------------------------------
kworker/0:1 14 615584.894138 1593194 120 615584.894138 193600.000000 1188160.000000
telnetd 75 615584.894138 579262 120 615584.894138 162720.000000 1203350.000000
find 434 615587.894138 1855 120 615587.894138 1100.000000 0.000000
R cat 435 615579.894138 1 120 615579.894138 10.000000 0.000000
[-- Attachment #5: proc_schedstat.log --]
[-- Type: text/x-log, Size: 95 bytes --]
version 15
timestamp 145532
cpu0 0 0 4718226 472 5453048 0 1715900000000 4121620000000 4717641
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: n_tty_write() going into schedule but NOT coming out
2013-03-30 12:35 n_tty_write() going into schedule but NOT coming out Vineet Gupta
@ 2013-04-01 13:57 ` Vineet Gupta
2013-04-01 15:10 ` Peter Hurley
2013-04-03 5:44 ` Ilya Zykov
1 sibling, 1 reply; 10+ messages in thread
From: Vineet Gupta @ 2013-04-01 13:57 UTC (permalink / raw)
To: lkml; +Cc: linux-serial, Ingo Molnar, Peter Zijlstra, Jiri Slaby,
Peter Hurley
Hi,
Any thoughts: I observe the same issue even with CONFIG_PREEMPT and
CONFIG_PREEMPT_COUNT
-Vineet
On 03/30/2013 06:05 PM, Vineet Gupta wrote:
> Hi,
>
> I've been stress testing ARC Linux 3.8 (same happens for 3.9-rc3 as well). The
> setup has 3 telnet sessions, each running find . -name "*" in a loop.
> The platform is a FPGA @ 80 MHz, running a single core ARC700 so kernel .config
> has !SMP and PREEMPT_NONE.
>
> After ~10 mins of run, I see that one of the telnet session gets stuck (and later
> the 2nd one as well), while system is still alive, 3rd telnet is running find merrily.
>
> [ARCLinux]$ ps
> ....
> 7 root 0:00 inetd
> 62 root 0:00 -/bin/sh
> 64 root 1:34 telnetd -i -l /bin/sh
> 65 root 0:00 /bin/sh
> 75 root 1:47 telnetd -i -l /bin/sh
> 76 root 0:00 /bin/sh
> 79 root 0:53 telnetd -i -l /bin/sh
> 80 root 0:00 /bin/sh
> 281 root 0:00 find / -name * <--- stuck
> 358 root 0:03 find / -name * <--- stuck
> 377 root 0:00 find / -name *
> 378 root 0:00 ps
>
> Hung find task is sitting in the schedule() call in n_tty_write()
>
> [ARCLinux]$ cat /proc/281/stack
> [<8065945e>] n_tty_write+0x23a/0x424
> [<80655cd4>] tty_write+0x1ac/0x2d4
> [<805976ba>] vfs_write+0x92/0x110
> [<80597816>] sys_write+0x4e/0x88
> [<8050e780>] ret_from_system_call+0x0/0x4
>
> This task never resumes out of schedule() - verified by putting a hardware
> breakpoint on next insn - using a JTAG host debugger.
>
> Attached are .config, /proc/281/sched, /proc/schedstat, /proc/sched_debug
>
> My knowledge of schedular is close to none, hence any tips to debug this further
> would be much appreciated.
>
>
> TIA,
> -Vineet
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: n_tty_write() going into schedule but NOT coming out
2013-04-01 13:57 ` Vineet Gupta
@ 2013-04-01 15:10 ` Peter Hurley
2013-04-02 11:09 ` Vineet Gupta
2013-04-02 13:26 ` Vineet Gupta
0 siblings, 2 replies; 10+ messages in thread
From: Peter Hurley @ 2013-04-01 15:10 UTC (permalink / raw)
To: Vineet Gupta; +Cc: lkml, linux-serial, Ingo Molnar, Peter Zijlstra, Jiri Slaby
On Mon, 2013-04-01 at 19:27 +0530, Vineet Gupta wrote:
> Hi,
>
> Any thoughts: I observe the same issue even with CONFIG_PREEMPT and
> CONFIG_PREEMPT_COUNT
>
> -Vineet
>
> On 03/30/2013 06:05 PM, Vineet Gupta wrote:
> > Hi,
> >
> > I've been stress testing ARC Linux 3.8 (same happens for 3.9-rc3 as well). The
> > setup has 3 telnet sessions, each running find . -name "*" in a loop.
> > The platform is a FPGA @ 80 MHz, running a single core ARC700 so kernel .config
> > has !SMP and PREEMPT_NONE.
> >
> > After ~10 mins of run, I see that one of the telnet session gets stuck (and later
> > the 2nd one as well), while system is still alive, 3rd telnet is running find merrily.
> >
> > [ARCLinux]$ ps
> > ....
> > 7 root 0:00 inetd
> > 62 root 0:00 -/bin/sh
> > 64 root 1:34 telnetd -i -l /bin/sh
> > 65 root 0:00 /bin/sh
> > 75 root 1:47 telnetd -i -l /bin/sh
> > 76 root 0:00 /bin/sh
> > 79 root 0:53 telnetd -i -l /bin/sh
> > 80 root 0:00 /bin/sh
> > 281 root 0:00 find / -name * <--- stuck
> > 358 root 0:03 find / -name * <--- stuck
> > 377 root 0:00 find / -name *
> > 378 root 0:00 ps
> >
> > Hung find task is sitting in the schedule() call in n_tty_write()
> >
> > [ARCLinux]$ cat /proc/281/stack
> > [<8065945e>] n_tty_write+0x23a/0x424
> > [<80655cd4>] tty_write+0x1ac/0x2d4
> > [<805976ba>] vfs_write+0x92/0x110
> > [<80597816>] sys_write+0x4e/0x88
> > [<8050e780>] ret_from_system_call+0x0/0x4
Likely the writer is stuck because the receive buffer is full and the
reader is hung. What are the respective shells and telnetd doing?
Regards,
Peter Hurley
PS - include the controlling tty column in your ps output :)
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: n_tty_write() going into schedule but NOT coming out
2013-04-01 15:10 ` Peter Hurley
@ 2013-04-02 11:09 ` Vineet Gupta
2013-04-02 13:26 ` Vineet Gupta
1 sibling, 0 replies; 10+ messages in thread
From: Vineet Gupta @ 2013-04-02 11:09 UTC (permalink / raw)
To: Peter Hurley; +Cc: lkml, linux-serial, Ingo Molnar, Peter Zijlstra, Jiri Slaby
Hi Peter,
On 04/01/2013 08:40 PM, Peter Hurley wrote:
> On Mon, 2013-04-01 at 19:27 +0530, Vineet Gupta wrote:
>> Hi,
>>
>> Any thoughts: I observe the same issue even with CONFIG_PREEMPT and
>> CONFIG_PREEMPT_COUNT
>>
>> -Vineet
>>
>> On 03/30/2013 06:05 PM, Vineet Gupta wrote:
>>> Hi,
>>>
>>> I've been stress testing ARC Linux 3.8 (same happens for 3.9-rc3 as well). The
>>> setup has 3 telnet sessions, each running find . -name "*" in a loop.
>>> The platform is a FPGA @ 80 MHz, running a single core ARC700 so kernel .config
>>> has !SMP and PREEMPT_NONE.
>>>
>>> After ~10 mins of run, I see that one of the telnet session gets stuck (and later
>>> the 2nd one as well), while system is still alive, 3rd telnet is running find merrily.
>>>
>>> [ARCLinux]$ ps
>>> ....
>>> 7 root 0:00 inetd
>>> 62 root 0:00 -/bin/sh
>>> 64 root 1:34 telnetd -i -l /bin/sh
>>> 65 root 0:00 /bin/sh
>>> 75 root 1:47 telnetd -i -l /bin/sh
>>> 76 root 0:00 /bin/sh
>>> 79 root 0:53 telnetd -i -l /bin/sh
>>> 80 root 0:00 /bin/sh
>>> 281 root 0:00 find / -name * <--- stuck
>>> 358 root 0:03 find / -name * <--- stuck
>>> 377 root 0:00 find / -name *
>>> 378 root 0:00 ps
>>>
>>> Hung find task is sitting in the schedule() call in n_tty_write()
>>>
>>> [ARCLinux]$ cat /proc/281/stack
>>> [<8065945e>] n_tty_write+0x23a/0x424
>>> [<80655cd4>] tty_write+0x1ac/0x2d4
>>> [<805976ba>] vfs_write+0x92/0x110
>>> [<80597816>] sys_write+0x4e/0x88
>>> [<8050e780>] ret_from_system_call+0x0/0x4
> Likely the writer is stuck because the receive buffer is full and the
> reader is hung. What are the respective shells and telnetd doing?
telnetd is in select syscall - waiting for new connections ?
shell is wait(2)ing - likely on the child "find" task.
/ # cat /proc/62/stack
[<8059e26a>] do_select+0x4b6/0x59c
[<8059e4f8>] core_sys_select+0x1a8/0x690
[<8059ea74>] sys_pselect6+0x94/0x254
[<80505fb4>] ret_from_system_call+0x0/0x4
/ # cat /proc/63/stack
[<80513a4a>] do_wait+0x166/0x188
[<80513ace>] sys_wait4+0x62/0xac
[<80505fb4>] ret_from_system_call+0x0/0x4
/ # cat /proc/445/stack
[<8064c7f2>] n_tty_write+0x23a/0x424
[<80649038>] tty_write+0x1a0/0x2b4
[<8058a4b6>] vfs_write+0x92/0x110
[<8058a612>] sys_write+0x4e/0x88
[<80505fb4>] ret_from_system_call+0x0/0x4
> PS - include the controlling tty column in your ps output :)
S UID PID PPID VSZ RSS TTY STIME TIME CMD
S 0 62 47 1768 488 0:0 16:01 00:01:44 telnetd -i -l /bin/sh
S 0 63 62 1768 512 pts2 16:01 00:00:00 /bin/sh
S 0 445 63 1840 488 pts2 16:30 00:00:00 find / -name *
Thx,
-Vineet
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: n_tty_write() going into schedule but NOT coming out
2013-04-01 15:10 ` Peter Hurley
2013-04-02 11:09 ` Vineet Gupta
@ 2013-04-02 13:26 ` Vineet Gupta
2013-04-02 13:38 ` Peter Hurley
2013-04-05 19:52 ` Peter Hurley
1 sibling, 2 replies; 10+ messages in thread
From: Vineet Gupta @ 2013-04-02 13:26 UTC (permalink / raw)
To: Peter Hurley; +Cc: lkml, linux-serial, Ingo Molnar, Peter Zijlstra, Jiri Slaby
On 04/01/2013 08:40 PM, Peter Hurley wrote:
> On Mon, 2013-04-01 at 19:27 +0530, Vineet Gupta wrote:
>> Hi,
>>
>> Any thoughts: I observe the same issue even with CONFIG_PREEMPT and
>> CONFIG_PREEMPT_COUNT
>>
>> -Vineet
>>
>> On 03/30/2013 06:05 PM, Vineet Gupta wrote:
>>> Hi,
>>>
>>> I've been stress testing ARC Linux 3.8 (same happens for 3.9-rc3 as well). The
>>> setup has 3 telnet sessions, each running find . -name "*" in a loop.
>>> The platform is a FPGA @ 80 MHz, running a single core ARC700 so kernel .config
>>> has !SMP and PREEMPT_NONE.
>>>
>>> After ~10 mins of run, I see that one of the telnet session gets stuck (and later
>>> the 2nd one as well), while system is still alive, 3rd telnet is running find merrily.
>>>
>>> [ARCLinux]$ ps
>>> ....
>>> 7 root 0:00 inetd
>>> 62 root 0:00 -/bin/sh
>>> 64 root 1:34 telnetd -i -l /bin/sh
>>> 65 root 0:00 /bin/sh
>>> 75 root 1:47 telnetd -i -l /bin/sh
>>> 76 root 0:00 /bin/sh
>>> 79 root 0:53 telnetd -i -l /bin/sh
>>> 80 root 0:00 /bin/sh
>>> 281 root 0:00 find / -name * <--- stuck
>>> 358 root 0:03 find / -name * <--- stuck
>>> 377 root 0:00 find / -name *
>>> 378 root 0:00 ps
>>>
>>> Hung find task is sitting in the schedule() call in n_tty_write()
>>>
>>> [ARCLinux]$ cat /proc/281/stack
>>> [<8065945e>] n_tty_write+0x23a/0x424
>>> [<80655cd4>] tty_write+0x1ac/0x2d4
>>> [<805976ba>] vfs_write+0x92/0x110
>>> [<80597816>] sys_write+0x4e/0x88
>>> [<8050e780>] ret_from_system_call+0x0/0x4
> Likely the writer is stuck because the receive buffer is full and the
> reader is hung. What are the respective shells and telnetd doing?
I've added a couple of printk's around that schedule() call and in one pathetic
case all 3 find tasks never return:
S 0 58 47 1768 488 0:0 00:00 00:03:50 telnetd -i -l /bin/sh
S 0 59 58 1768 520 pts0 00:00 00:00:01 /bin/sh
S 0 61 47 1768 488 0:0 00:00 00:03:45 telnetd -i -l /bin/sh
S 0 62 61 1768 528 pts1 00:00 00:00:01 /bin/sh
S 0 65 47 1768 488 0:0 00:00 00:03:23 telnetd -i -l /bin/sh
S 0 66 65 1776 528 pts2 00:00 00:00:01 /bin/sh
S 0 833 66 1840 488 pts2 00:58 00:00:01 find / -name * <--
S 0 871 62 1832 480 pts1 01:01 00:00:00 find / -name * <--
S 0 881 59 1840 488 pts0 01:01 00:00:02 find / -name * <--
---> 833 8e21c580
<--- 833 8e21c580
---> 833 8e21c580
---> 871 8e21c840
<--- 871 8e21c840
---> 871 8e21c840
---> 881 8e21c2c0
<--- 881 8e21c2c0
---> 881 8e21c2c0
I don't undersand how the receive buffer full/empty is coming into play - that
schedule() call is not a wait queue or something, it's an unconditional yield,
expecting sched to unconditionally return at some point - unless ofcourse - that
itself relies on some buffer r/w ISR semantics triggering an ISR, which makes the
task runnable again causing schdule() to return.
-Vineet
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: n_tty_write() going into schedule but NOT coming out
2013-04-02 13:26 ` Vineet Gupta
@ 2013-04-02 13:38 ` Peter Hurley
2013-04-05 19:52 ` Peter Hurley
1 sibling, 0 replies; 10+ messages in thread
From: Peter Hurley @ 2013-04-02 13:38 UTC (permalink / raw)
To: Vineet Gupta; +Cc: lkml, linux-serial, Ingo Molnar, Peter Zijlstra, Jiri Slaby
On Tue, 2013-04-02 at 18:56 +0530, Vineet Gupta wrote:
> On 04/01/2013 08:40 PM, Peter Hurley wrote:
> > On Mon, 2013-04-01 at 19:27 +0530, Vineet Gupta wrote:
> >> On 03/30/2013 06:05 PM, Vineet Gupta wrote:
> >>> Hung find task is sitting in the schedule() call in n_tty_write()
> >>>
> >>> [ARCLinux]$ cat /proc/281/stack
> >>> [<8065945e>] n_tty_write+0x23a/0x424
> >>> [<80655cd4>] tty_write+0x1ac/0x2d4
> >>> [<805976ba>] vfs_write+0x92/0x110
> >>> [<80597816>] sys_write+0x4e/0x88
> >>> [<8050e780>] ret_from_system_call+0x0/0x4
> > Likely the writer is stuck because the receive buffer is full and the
> > reader is hung. What are the respective shells and telnetd doing?
>
> I don't undersand how the receive buffer full/empty is coming into play - that
> schedule() call is not a wait queue or something
If n_tty_write() blocks [the schedule() call], then it is on a wait
queue (tty->write_wait) which is woken by pty's unthrottle() called from
n_tty_read from the linked pty when space becomes available in the
receive buffer.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: n_tty_write() going into schedule but NOT coming out
2013-03-30 12:35 n_tty_write() going into schedule but NOT coming out Vineet Gupta
2013-04-01 13:57 ` Vineet Gupta
@ 2013-04-03 5:44 ` Ilya Zykov
2013-04-03 7:42 ` Vineet Gupta
1 sibling, 1 reply; 10+ messages in thread
From: Ilya Zykov @ 2013-04-03 5:44 UTC (permalink / raw)
To: Vineet Gupta
Cc: lkml, linux-serial, Ingo Molnar, Peter Zijlstra, Jiri Slaby,
Peter Hurley
Please, try this patch maybe it can help localize your problem.
drivers/tty/n_tty.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index 05e72be..28f15d0 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -230,6 +230,7 @@ static void reset_buffer_flags(struct tty_struct *tty)
ldata->canon_head = ldata->canon_data = ldata->erasing = 0;
bitmap_zero(ldata->read_flags, N_TTY_BUF_SIZE);
n_tty_set_room(tty);
+ check_unthrottle(tty);
}
/**
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: n_tty_write() going into schedule but NOT coming out
2013-04-03 5:44 ` Ilya Zykov
@ 2013-04-03 7:42 ` Vineet Gupta
0 siblings, 0 replies; 10+ messages in thread
From: Vineet Gupta @ 2013-04-03 7:42 UTC (permalink / raw)
To: Ilya Zykov
Cc: lkml, linux-serial, Ingo Molnar, Peter Zijlstra, Jiri Slaby,
Peter Hurley
Hi Ilya
On 04/03/2013 11:14 AM, Ilya Zykov wrote:
> Please, try this patch maybe it can help localize your problem.
>
> drivers/tty/n_tty.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> index 05e72be..28f15d0 100644
> --- a/drivers/tty/n_tty.c
> +++ b/drivers/tty/n_tty.c
> @@ -230,6 +230,7 @@ static void reset_buffer_flags(struct tty_struct *tty)
> ldata->canon_head = ldata->canon_data = ldata->erasing = 0;
> bitmap_zero(ldata->read_flags, N_TTY_BUF_SIZE);
> n_tty_set_room(tty);
> + check_unthrottle(tty);
> }
>
> /**
Thanks for the patch - but it didn't help at all - I see the same lockup.
I'm no tty expert so don't understand how this patch would help - care to elaborate ?
Thx,
-Vineet
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: n_tty_write() going into schedule but NOT coming out
2013-04-02 13:26 ` Vineet Gupta
2013-04-02 13:38 ` Peter Hurley
@ 2013-04-05 19:52 ` Peter Hurley
2013-04-06 9:32 ` Vineet Gupta
1 sibling, 1 reply; 10+ messages in thread
From: Peter Hurley @ 2013-04-05 19:52 UTC (permalink / raw)
To: Vineet Gupta; +Cc: lkml, linux-serial, Jiri Slaby
[ --cc Ingo and Peter Z. as this is not scheduler-related]
On Tue, 2013-04-02 at 18:56 +0530, Vineet Gupta wrote:
> On 04/01/2013 08:40 PM, Peter Hurley wrote:
> > On Mon, 2013-04-01 at 19:27 +0530, Vineet Gupta wrote:
> >> Hi,
> >>
> >> Any thoughts: I observe the same issue even with CONFIG_PREEMPT and
> >> CONFIG_PREEMPT_COUNT
I'll see if I can reproduce this over the weekend on an old single-core
laptop I still have.
There were some race conditions in the N_TTY line discipline which I
recently fixed. Those changes are in linux-next. Can you test if this is
reproducible on linux-next?
Assuming I don't reproduce this on the laptop, the only other
explanation I can think of right now is that ARCLinux is not properly
handling signal-driven i/o (assuming the BusyBox /bin/sh uses SIGIO). Do
you know if there is anything special about the way ARCLinux handle
signals?
Regards,
Peter Hurley
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: n_tty_write() going into schedule but NOT coming out
2013-04-05 19:52 ` Peter Hurley
@ 2013-04-06 9:32 ` Vineet Gupta
0 siblings, 0 replies; 10+ messages in thread
From: Vineet Gupta @ 2013-04-06 9:32 UTC (permalink / raw)
To: Peter Hurley; +Cc: lkml, linux-serial, Jiri Slaby
On 04/06/2013 01:22 AM, Peter Hurley wrote:
> I'll see if I can reproduce this over the weekend on an old single-core laptop I
> still have.
TIA for doing this.
> There were some race conditions in the N_TTY line discipline which I
> recently fixed. Those changes are in linux-next. Can you test if this is
> reproducible on linux-next?
I tried today's -next and I see the same issue :-(
> Assuming I don't reproduce this on the laptop, the only other
> explanation I can think of right now is that ARCLinux is not properly
> handling signal-driven i/o (assuming the BusyBox /bin/sh uses SIGIO). Do
> you know if there is anything special about the way ARCLinux handle
> signals?
No, it's pretty standard stuff - we are uClibc based though - as opposed to glibc
so there might be some subtleties - but we do run LTP open posix regularly. Also
the test setup is a slowish 80MHz FPGA so this is not something many people have
in their regular test setups.
I'll start reading the relevant code myself - and will be willing to take any
debug/test patches which help with troubleshooting of this issue.
Just to re-iterate, my test setup has a minimal busybox based rootfs, 3 telnet
sessions, each running a
while true; do find / -name "*" ; done
I'm running out of ramfs, no external disk/nfs mounts to reduce the peripheral I/O
slowing down the find (although NFS stuff will be caches anyways after first fetch).
Also please make sure you have CONFIG_PREEMPT_COUNT otherwise there's a
possibility (atleast on ARC builds) that a stale register causes timer list to be
corrupted in mod_timer().
Thx,
-Vineet
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2013-04-06 9:32 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-03-30 12:35 n_tty_write() going into schedule but NOT coming out Vineet Gupta
2013-04-01 13:57 ` Vineet Gupta
2013-04-01 15:10 ` Peter Hurley
2013-04-02 11:09 ` Vineet Gupta
2013-04-02 13:26 ` Vineet Gupta
2013-04-02 13:38 ` Peter Hurley
2013-04-05 19:52 ` Peter Hurley
2013-04-06 9:32 ` Vineet Gupta
2013-04-03 5:44 ` Ilya Zykov
2013-04-03 7:42 ` Vineet Gupta
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).