* System freezes with high network activity @ 2011-12-02 16:44 Jose Luis Salas 2011-12-02 19:43 ` john stultz 0 siblings, 1 reply; 7+ messages in thread From: Jose Luis Salas @ 2011-12-02 16:44 UTC (permalink / raw) To: linux-kernel; +Cc: John Stultz, Thomas Gleixner, Jonathan Nieder Hello there, I' ve filled a bug against linux kernel in Debian [1] for the linux-kernel-image 2.6.32. The problem is that the system freezes ( no complete ) when high network activity occurrs, for example copying files over NFS or downloading files with Bittorrent. When the system is frozen, there is no network connecivity, the console freezes ( with top program, for ex. ) until a key from the laptop is pressed or when network activity drops. The network throughtput also drops before the freeze. I've also realized that clock drifts ( it's frozen too ) when the system is frozen. This problem didn't occur in Linux kernel 2.6.26 ( from the Lenny release ), I think >2.6.26 are affected, but I need to confirm it ( >= 2.6.30 are included for sure ). After investigating for a while ( tested noapic, noacpi and several switches with no luck.) , I've search for a solution and I finally found it. I think that the bug is caused by changes in the clocksource between kernels and the powernow-k8 module. One time I tested booting the xen-patched the linux kernel and the problem dissapeared, the net worked at full speed but there was no cpu scaling. Now, with recent kernels, I boot the kernel with the clocksource=jiffies option and the powersaved daemon to do the frequency scaling, the net works al full speed. With acpi_pm and tsc options doesn't work neither. More information as acpidump output and dmesg are included in the [1] Debian bug too. Thanks. [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=583363 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: System freezes with high network activity 2011-12-02 16:44 System freezes with high network activity Jose Luis Salas @ 2011-12-02 19:43 ` john stultz 2011-12-02 20:54 ` Jose Luis Salas 0 siblings, 1 reply; 7+ messages in thread From: john stultz @ 2011-12-02 19:43 UTC (permalink / raw) To: Jose Luis Salas; +Cc: linux-kernel, Thomas Gleixner, Jonathan Nieder On Fri, 2011-12-02 at 17:44 +0100, Jose Luis Salas wrote: > Hello there, > > I' ve filled a bug against linux kernel in Debian [1] for the > linux-kernel-image 2.6.32. > > The problem is that the system freezes ( no complete ) when high > network activity occurrs, for example copying files over NFS or > downloading files with Bittorrent. > > When the system is frozen, there is no network connecivity, the > console freezes ( with top program, for ex. ) until a key from the > laptop is pressed or when network activity drops. The network > throughtput also drops before the freeze. > > I've also realized that clock drifts ( it's frozen too ) when the > system is frozen. > > This problem didn't occur in Linux kernel 2.6.26 ( from the Lenny > release ), I think >2.6.26 are affected, but I need to confirm it ( >= > 2.6.30 are included for sure ). > > After investigating for a while ( tested noapic, noacpi and several > switches with no luck.) , I've search for a solution and I finally > found it. I think that the bug is caused by changes in the > clocksource between kernels and the powernow-k8 module. > > One time I tested booting the xen-patched the linux kernel and the > problem dissapeared, the net worked at full speed but there was no cpu > scaling. > > Now, with recent kernels, I boot the kernel with the > clocksource=jiffies option and the powersaved daemon to do the > frequency scaling, the net works al full speed. With acpi_pm and tsc > options doesn't work neither. So, could you attach a full dmesg when you're not using clocksource=jiffies? Also, when not using clocksource=jiffies, does booting with nohz=off change the behavior? thanks -john ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: System freezes with high network activity 2011-12-02 19:43 ` john stultz @ 2011-12-02 20:54 ` Jose Luis Salas 2011-12-02 22:28 ` john stultz 0 siblings, 1 reply; 7+ messages in thread From: Jose Luis Salas @ 2011-12-02 20:54 UTC (permalink / raw) To: john stultz; +Cc: linux-kernel, Thomas Gleixner, Jonathan Nieder [-- Attachment #1: Type: text/plain, Size: 422 bytes --] Hi, attached is the dmesg without the clocksource option. the nohz=off *seems* to avoid the problem, I'm testing with NFS and Iperf now. Thanks. On Fri, Dec 2, 2011 at 8:43 PM, john stultz <johnstul@us.ibm.com> wrote: > So, could you attach a full dmesg when you're not using > clocksource=jiffies? > > Also, when not using clocksource=jiffies, does booting with nohz=off > change the behavior? > > thanks > -john > > [-- Attachment #2: dmesg-without-jiffies.txt --] [-- Type: text/plain, Size: 33138 bytes --] [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 2.6.32-5-686 (Debian 2.6.32-35) (dannf@debian.org) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Mon Jun 13 04:13:06 UTC 2011 [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] NSC Geode by NSC [ 0.000000] Cyrix CyrixInstead [ 0.000000] Centaur CentaurHauls [ 0.000000] Transmeta GenuineTMx86 [ 0.000000] Transmeta TransmetaCPU [ 0.000000] UMC UMC UMC UMC [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009f400 (usable) [ 0.000000] BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved) [ 0.000000] BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved) [ 0.000000] BIOS-e820: 0000000000100000 - 000000003def0000 (usable) [ 0.000000] BIOS-e820: 000000003def0000 - 000000003defa000 (ACPI data) [ 0.000000] BIOS-e820: 000000003defa000 - 000000003df00000 (ACPI NVS) [ 0.000000] BIOS-e820: 000000003df00000 - 0000000040000000 (reserved) [ 0.000000] BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved) [ 0.000000] DMI present. [ 0.000000] last_pfn = 0x3def0 max_arch_pfn = 0x100000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-CBFFF write-protect [ 0.000000] CC000-DBFFF uncachable [ 0.000000] DC000-E3FFF write-back [ 0.000000] E4000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000000000 mask FFC0000000 write-back [ 0.000000] 1 base 003E000000 mask FFFE000000 uncachable [ 0.000000] 2 base 00E0000000 mask FFFE000000 write-combining [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] initial memory mapped : 0 - 01800000 [ 0.000000] init_memory_mapping: 0000000000000000-00000000373fe000 [ 0.000000] 0000000000 - 0000400000 page 4k [ 0.000000] 0000400000 - 0037000000 page 2M [ 0.000000] 0037000000 - 00373fe000 page 4k [ 0.000000] kernel direct mapping tables up to 373fe000 @ 7000-d000 [ 0.000000] RAMDISK: 36f50000 - 3779f668 [ 0.000000] Allocated new RAMDISK: 00100000 - 0094f668 [ 0.000000] Move RAMDISK from 0000000036f50000 - 000000003779f667 to 00100000 - 0094f667 [ 0.000000] ACPI: RSDP 000f7f60 00014 (v00 PTLTD ) [ 0.000000] ACPI: RSDT 3def619d 00034 (v01 PTLTD RSDT 06040000 LTP 00000000) [ 0.000000] ACPI: FACP 3def9e3e 00074 (v01 SiS 755F 06040000 PTL 000F4240) [ 0.000000] ACPI: DSDT 3def61d1 03C6D (v01 PTLTD 755 06040000 MSFT 0100000E) [ 0.000000] ACPI: FACS 3defafc0 00040 [ 0.000000] ACPI: SSDT 3def9eb2 000D6 (v01 PTLTD POWERNOW 06040000 LTP 00000001) [ 0.000000] ACPI: APIC 3def9f88 00050 (v01 PTLTD ? APIC 06040000 LTP 00000000) [ 0.000000] ACPI: BOOT 3def9fd8 00028 (v01 PTLTD $SBFTBL$ 06040000 LTP 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] 106MB HIGHMEM available. [ 0.000000] 883MB LOWMEM available. [ 0.000000] mapped low ram: 0 - 373fe000 [ 0.000000] low ram: 0 - 373fe000 [ 0.000000] node 0 low ram: 00000000 - 373fe000 [ 0.000000] node 0 bootmap 00009000 - 0000fe80 [ 0.000000] (9 early reservations) ==> bootmem [0000000000 - 00373fe000] [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] [ 0.000000] #1 [0000001000 - 0000002000] EX TRAMPOLINE ==> [0000001000 - 0000002000] [ 0.000000] #2 [0000006000 - 0000007000] TRAMPOLINE ==> [0000006000 - 0000007000] [ 0.000000] #3 [0001000000 - 00014c8bb4] TEXT DATA BSS ==> [0001000000 - 00014c8bb4] [ 0.000000] #4 [000009f400 - 0000100000] BIOS reserved ==> [000009f400 - 0000100000] [ 0.000000] #5 [00014c9000 - 00014cf190] BRK ==> [00014c9000 - 00014cf190] [ 0.000000] #6 [0000007000 - 0000009000] PGTABLE ==> [0000007000 - 0000009000] [ 0.000000] #7 [0000100000 - 000094f668] NEW RAMDISK ==> [0000100000 - 000094f668] [ 0.000000] #8 [0000009000 - 0000010000] BOOTMAP ==> [0000009000 - 0000010000] [ 0.000000] found SMP MP-table at [c00f7fd0] f7fd0 [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000000 -> 0x00001000 [ 0.000000] Normal 0x00001000 -> 0x000373fe [ 0.000000] HighMem 0x000373fe -> 0x0003def0 [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000000 -> 0x0000009f [ 0.000000] 0: 0x00000100 -> 0x0003def0 [ 0.000000] On node 0 totalpages: 253583 [ 0.000000] free_area_init_node: node 0, pgdat c13b08a0, node_mem_map c14d1000 [ 0.000000] DMA zone: 32 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 3967 pages, LIFO batch:0 [ 0.000000] Normal zone: 1736 pages used for memmap [ 0.000000] Normal zone: 220470 pages, LIFO batch:31 [ 0.000000] HighMem zone: 214 pages used for memmap [ 0.000000] HighMem zone: 27164 pages, LIFO batch:7 [ 0.000000] Using APIC driver default [ 0.000000] ACPI: PM-Timer IO Port: 0x8008 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ2 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs [ 0.000000] nr_irqs_gsi: 24 [ 0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000dc000 [ 0.000000] PM: Registered nosave memory: 00000000000dc000 - 0000000000100000 [ 0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:bff00000) [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 14 pages/cpu @c2000000 s34328 r0 d23016 u4194304 [ 0.000000] pcpu-alloc: s34328 r0 d23016 u4194304 alloc=1*4194304 [ 0.000000] pcpu-alloc: [0] 0 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 251601 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.32-5-686 root=UUID=b2b271f0-0002-47e5-91fb-dc897e5ade90 ro ide_core.nohpa=0.0 libata.ignore_hpa=1 [ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes) [ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) [ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) [ 0.000000] Enabling fast FPU save and restore... done. [ 0.000000] Enabling unmasked SIMD FPU exception support... done. [ 0.000000] Initializing CPU#0 [ 0.000000] Initializing HighMem for node 0 (000373fe:0003def0) [ 0.000000] Memory: 991696k/1014720k available (2504k kernel code, 22280k reserved, 1313k data, 380k init, 109512k highmem) [ 0.000000] virtual kernel memory layout: [ 0.000000] fixmap : 0xffd56000 - 0xfffff000 (2724 kB) [ 0.000000] pkmap : 0xff400000 - 0xff800000 (4096 kB) [ 0.000000] vmalloc : 0xf7bfe000 - 0xff3fe000 ( 120 MB) [ 0.000000] lowmem : 0xc0000000 - 0xf73fe000 ( 883 MB) [ 0.000000] .init : 0xc13bb000 - 0xc141a000 ( 380 kB) [ 0.000000] .data : 0xc1272001 - 0xc13ba4c0 (1313 kB) [ 0.000000] .text : 0xc1000000 - 0xc1272001 (2504 kB) [ 0.000000] Checking if this processor honours the WP bit even in supervisor mode...Ok. [ 0.000000] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] NR_IRQS:1280 [ 0.000000] spurious 8259A interrupt: IRQ7. [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] Fast TSC calibration using PIT [ 0.000000] Detected 1800.182 MHz processor. [ 0.008006] Calibrating delay loop (skipped), value calculated using timer frequency.. 3600.36 BogoMIPS (lpj=7200728) [ 0.008085] Security Framework initialized [ 0.008121] SELinux: Disabled at boot. [ 0.008157] Mount-cache hash table entries: 512 [ 0.008324] Initializing cgroup subsys ns [ 0.008355] Initializing cgroup subsys cpuacct [ 0.008387] Initializing cgroup subsys devices [ 0.008417] Initializing cgroup subsys freezer [ 0.008447] Initializing cgroup subsys net_cls [ 0.008499] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) [ 0.008531] CPU: L2 Cache: 128K (64 bytes/line) [ 0.008563] mce: CPU supports 5 MCE banks [ 0.008612] Performance Events: AMD PMU driver. [ 0.008670] ... version: 0 [ 0.008699] ... bit width: 48 [ 0.008728] ... generic registers: 4 [ 0.008757] ... value mask: 0000ffffffffffff [ 0.008787] ... max period: 00007fffffffffff [ 0.008817] ... fixed-purpose events: 0 [ 0.008846] ... event mask: 000000000000000f [ 0.008877] Checking 'hlt' instruction... OK. [ 0.024358] SMP alternatives: switching to UP code [ 0.028273] Freeing SMP alternatives: 13k freed [ 0.028323] ACPI: Core revision 20090903 [ 0.036090] Enabling APIC mode: Flat. Using 1 I/O APICs [ 0.040249] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.079962] CPU0: Mobile AMD Sempron(tm) Processor 3000+ stepping 02 [ 0.084001] Brought up 1 CPUs [ 0.084001] Total of 1 processors activated (3600.36 BogoMIPS). [ 0.084001] CPU0 attaching NULL sched-domain. [ 0.084001] devtmpfs: initialized [ 0.084001] regulator: core version 0.5 [ 0.084001] NET: Registered protocol family 16 [ 0.084001] ACPI: bus type pci registered [ 0.084001] PCI: PCI BIOS revision 2.10 entry at 0xfd776, last bus=1 [ 0.084001] PCI: Using configuration type 1 for base access [ 0.084001] bio: create slab <bio-0> at 0 [ 0.084001] ACPI: EC: Look up EC in DSDT [ 0.096054] ACPI: Interpreter enabled [ 0.096091] ACPI: (supports S0 S3 S4 S5) [ 0.096229] ACPI: Using IOAPIC for interrupt routing [ 0.101725] ACPI: EC: GPE = 0x19, I/O: command/status = 0x66, data = 0x62 [ 0.101891] ACPI: No dock devices found. [ 0.102950] ACPI: PCI Root Bridge [PCI0] (0000:00) [ 0.103028] pci 0000:00:00.0: reg 10 32bit mmio: [0xe0000000-0xe1ffffff] [ 0.103125] pci 0000:00:02.0: Enabling SiS 96x SMBus [ 0.103183] pci 0000:00:02.1: reg 20 io port: [0x8100-0x811f] [ 0.103210] pci 0000:00:02.5: reg 10 io port: [0x1f0-0x1f7] [ 0.103216] pci 0000:00:02.5: reg 14 io port: [0x3f4-0x3f7] [ 0.103222] pci 0000:00:02.5: reg 18 io port: [0x170-0x177] [ 0.103227] pci 0000:00:02.5: reg 1c io port: [0x374-0x377] [ 0.103233] pci 0000:00:02.5: reg 20 io port: [0x2000-0x200f] [ 0.103264] pci 0000:00:02.6: reg 10 io port: [0x1000-0x10ff] [ 0.103270] pci 0000:00:02.6: reg 14 io port: [0x1c00-0x1c7f] [ 0.103297] pci 0000:00:02.6: supports D1 D2 [ 0.103300] pci 0000:00:02.6: PME# supported from D3hot D3cold [ 0.103332] pci 0000:00:02.6: PME# disabled [ 0.103386] pci 0000:00:02.7: reg 10 io port: [0x1400-0x14ff] [ 0.103392] pci 0000:00:02.7: reg 14 io port: [0x1c80-0x1cff] [ 0.103419] pci 0000:00:02.7: supports D1 D2 [ 0.103421] pci 0000:00:02.7: PME# supported from D3hot D3cold [ 0.103453] pci 0000:00:02.7: PME# disabled [ 0.103497] pci 0000:00:03.0: reg 10 32bit mmio: [0xe2002000-0xe2002fff] [ 0.103532] pci 0000:00:03.1: reg 10 32bit mmio: [0xe2003000-0xe2003fff] [ 0.103572] pci 0000:00:03.2: reg 10 32bit mmio: [0xe2004000-0xe2004fff] [ 0.103599] pci 0000:00:03.2: PME# supported from D0 D3hot D3cold [ 0.103631] pci 0000:00:03.2: PME# disabled [ 0.103686] pci 0000:00:04.0: reg 10 io port: [0x1800-0x18ff] [ 0.103692] pci 0000:00:04.0: reg 14 32bit mmio: [0xe2005000-0xe2005fff] [ 0.103710] pci 0000:00:04.0: reg 30 32bit mmio pref: [0x000000-0x01ffff] [ 0.103723] pci 0000:00:04.0: supports D1 D2 [ 0.103725] pci 0000:00:04.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.103757] pci 0000:00:04.0: PME# disabled [ 0.103810] pci 0000:00:06.0: reg 10 32bit mmio: [0x000000-0x000fff] [ 0.103824] pci 0000:00:06.0: supports D1 D2 [ 0.103827] pci 0000:00:06.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.103859] pci 0000:00:06.0: PME# disabled [ 0.103906] pci 0000:00:0b.0: reg 10 32bit mmio: [0xe2000000-0xe2001fff] [ 0.104045] pci 0000:01:00.0: reg 10 32bit mmio pref: [0xe8000000-0xefffffff] [ 0.104050] pci 0000:01:00.0: reg 14 32bit mmio: [0xe2100000-0xe211ffff] [ 0.104055] pci 0000:01:00.0: reg 18 io port: [0xa000-0xa07f] [ 0.104070] pci 0000:01:00.0: supports D1 D2 [ 0.104092] pci 0000:00:01.0: bridge io port: [0xa000-0xafff] [ 0.104096] pci 0000:00:01.0: bridge 32bit mmio: [0xe2100000-0xe21fffff] [ 0.104101] pci 0000:00:01.0: bridge 32bit mmio pref: [0xe8000000-0xefffffff] [ 0.104119] pci_bus 0000:00: on NUMA node 0 [ 0.104124] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] [ 0.111159] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 *7 9 10 11) [ 0.111498] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 *4 5 7 9 10 11) [ 0.111822] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 7 9 10 11) [ 0.112161] ACPI: PCI Interrupt Link [LNKD] (IRQs *3 4 5 7 9 10 11) [ 0.112506] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 7 *9 10 11) [ 0.112830] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 7 9 10 *11) [ 0.113149] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 7 9 10 11) *0, disabled. [ 0.113519] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 7 9 *10 11) [ 0.113887] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.113922] vgaarb: loaded [ 0.114044] PCI: Using ACPI for IRQ routing [ 0.114247] Switching to clocksource tsc [ 0.115699] pnp: PnP ACPI init [ 0.115761] ACPI: bus type pnp registered [ 0.116000] pnp: PnP ACPI: found 8 devices [ 0.116000] ACPI: ACPI bus type pnp unregistered [ 0.116000] PnPBIOS: Disabled by ACPI PNP [ 0.116000] system 00:04: ioport range 0x8000-0x807f has been reserved [ 0.116000] system 00:04: ioport range 0x8080-0x80ff has been reserved [ 0.116000] system 00:04: ioport range 0x8100-0x811f has been reserved [ 0.116000] system 00:04: ioport range 0x4d0-0x4d1 has been reserved [ 0.116000] system 00:04: ioport range 0x3f0-0x3f1 has been reserved [ 0.116000] system 00:04: iomem range 0xfec00000-0xfecfffff could not be reserved [ 0.116000] system 00:04: iomem range 0xfed00000-0xfed003ff has been reserved [ 0.116000] system 00:04: iomem range 0xfee00000-0xfeefffff has been reserved [ 0.116000] system 00:04: iomem range 0xffc00000-0xffc00fff has been reserved [ 0.116000] system 00:04: iomem range 0xffe00000-0xffe00fff has been reserved [ 0.116000] system 00:04: iomem range 0xffe80000-0xffefffff has been reserved [ 0.116000] system 00:04: iomem range 0xfffe0000-0xfffeffff has been reserved [ 0.116004] system 00:04: iomem range 0xffff0000-0xffffffff has been reserved [ 0.150797] pci 0000:00:01.0: PCI bridge, secondary bus 0000:01 [ 0.150831] pci 0000:00:01.0: IO window: 0xa000-0xafff [ 0.150863] pci 0000:00:01.0: MEM window: 0xe2100000-0xe21fffff [ 0.150896] pci 0000:00:01.0: PREFETCH window: 0xe8000000-0xefffffff [ 0.150930] pci 0000:00:06.0: CardBus bridge, secondary bus 0000:02 [ 0.150961] pci 0000:00:06.0: IO window: 0x002400-0x0024ff [ 0.150992] pci 0000:00:06.0: IO window: 0x002800-0x0028ff [ 0.151024] pci 0000:00:06.0: PREFETCH window: 0x40000000-0x43ffffff [ 0.151056] pci 0000:00:06.0: MEM window: 0x44000000-0x47ffffff [ 0.151099] pci 0000:00:06.0: enabling device (0000 -> 0003) [ 0.151139] pci 0000:00:06.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19 [ 0.151174] pci_bus 0000:00: resource 0 io: [0x00-0xffff] [ 0.151177] pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffff] [ 0.151181] pci_bus 0000:01: resource 0 io: [0xa000-0xafff] [ 0.151184] pci_bus 0000:01: resource 1 mem: [0xe2100000-0xe21fffff] [ 0.151187] pci_bus 0000:01: resource 2 pref mem [0xe8000000-0xefffffff] [ 0.151191] pci_bus 0000:02: resource 0 io: [0x2400-0x24ff] [ 0.151194] pci_bus 0000:02: resource 1 io: [0x2800-0x28ff] [ 0.151197] pci_bus 0000:02: resource 2 pref mem [0x40000000-0x43ffffff] [ 0.151201] pci_bus 0000:02: resource 3 mem: [0x44000000-0x47ffffff] [ 0.151241] NET: Registered protocol family 2 [ 0.151368] IP route cache hash table entries: 32768 (order: 5, 131072 bytes) [ 0.151842] TCP established hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.153096] TCP bind hash table entries: 65536 (order: 7, 524288 bytes) [ 0.153783] TCP: Hash tables configured (established 131072 bind 65536) [ 0.153816] TCP reno registered [ 0.153961] NET: Registered protocol family 1 [ 0.582296] pci 0000:01:00.0: Boot video device [ 0.582357] Unpacking initramfs... [ 0.849451] Freeing initrd memory: 8509k freed [ 0.859652] Simple Boot Flag at 0x38 set to 0x1 [ 0.859971] audit: initializing netlink socket (disabled) [ 0.860018] type=2000 audit(1322856932.857:1): initialized [ 0.863795] highmem bounce pool size: 64 pages [ 0.863831] HugeTLB registered 4 MB page size, pre-allocated 0 pages [ 0.865414] VFS: Disk quotas dquot_6.5.2 [ 0.865522] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) [ 0.865629] msgmni has been set to 1740 [ 0.865922] alg: No test for stdrng (krng) [ 0.866023] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) [ 0.866057] io scheduler noop registered [ 0.866087] io scheduler anticipatory registered [ 0.866116] io scheduler deadline registered [ 0.866182] io scheduler cfq registered (default) [ 0.866385] isapnp: Scanning for PnP cards... [ 1.219694] isapnp: No Plug & Play device found [ 1.221203] Linux agpgart interface v0.103 [ 1.221289] agpgart-amd64 0000:00:00.0: AGP bridge [1039/0760] [ 1.222639] agpgart-amd64 0000:00:00.0: AGP aperture is 32M @ 0xe0000000 [ 1.222768] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 1.223091] serial 0000:00:02.6: PCI INT C -> GSI 18 (level, low) -> IRQ 18 [ 1.223126] serial 0000:00:02.6: PCI INT C disabled [ 1.223247] PNP: PS/2 Controller [PNP0303:KBC,PNP0f13:PS2M] at 0x60,0x64 irq 1,12 [ 1.225482] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.225517] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.225660] mice: PS/2 mouse device common for all mice [ 1.225750] rtc_cmos 00:02: RTC can wake from S4 [ 1.225820] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0 [ 1.225873] rtc0: alarms up to one year, y3k, 114 bytes nvram [ 1.225917] cpuidle: using governor ladder [ 1.225947] cpuidle: using governor menu [ 1.225979] No iBFT detected. [ 1.226412] TCP cubic registered [ 1.226591] NET: Registered protocol family 10 [ 1.227452] Mobile IPv6 [ 1.227482] NET: Registered protocol family 17 [ 1.227533] Using IPI No-Shortcut mode [ 1.227667] PM: Resume from disk failed. [ 1.227685] registered taskstats version 1 [ 1.227917] rtc_cmos 00:02: setting system clock to 2011-12-02 20:15:34 UTC (1322856934) [ 1.227980] Initalizing network drop monitor service [ 1.228042] Freeing unused kernel memory: 380k freed [ 1.228531] Write protecting the kernel text: 2508k [ 1.228614] Write protecting the kernel read-only data: 904k [ 1.250534] <30>udevd[47]: starting version 172 [ 1.257017] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 1.454602] SCSI subsystem initialized [ 1.483818] sis900.c: v1.08.10 Apr. 2 2006 [ 1.483897] sis900 0000:00:04.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19 [ 1.488579] usbcore: registered new interface driver usbfs [ 1.488826] usbcore: registered new interface driver hub [ 1.489107] usbcore: registered new device driver usb [ 1.499745] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.500124] 0000:00:04.0: Realtek RTL8201 PHY transceiver found at address 13. [ 1.505691] 0000:00:04.0: Using transceiver found at address 13 as default [ 1.532562] sisfb: Video ROM found [ 1.533330] sisfb: Video RAM at 0xe8000000, mapped to 0xf7e80000, size 32768k [ 1.533362] sisfb: MMIO at 0xe2100000, mapped to 0xf7e40000, size 128k [ 1.533393] sisfb: Memory heap starting at 32160K, size 32K [ 1.533439] sisfb: Detected SiS302LV video bridge [ 1.593962] sisfb: Detected 1024x768 flat panel [ 1.593966] sisfb: Detected LCD PDC1 0x00 (for LCD=CRT1) [ 1.672565] sisfb: CRT1 DDC supported [ 1.672595] sisfb: CRT1 DDC level: 2 [ 1.826616] sisfb: Monitor range H 24-83KHz, V 50-76Hz, Max. dotclock 170MHz [ 1.826654] sisfb: Default mode is 1024x768x16 (75Hz) [ 1.826689] sisfb: Initial vbflags 0x8000012 [ 1.827827] eth0: SiS 900 PCI Fast Ethernet at 0x1800, IRQ 19, 00:16:36:04:cf:d6 [ 1.827918] ehci_hcd 0000:00:03.2: PCI INT D -> GSI 23 (level, low) -> IRQ 23 [ 1.827974] ehci_hcd 0000:00:03.2: EHCI Host Controller [ 1.828061] ehci_hcd 0000:00:03.2: new USB bus registered, assigned bus number 1 [ 1.828137] ehci_hcd 0000:00:03.2: cache line size of 64 is not supported [ 1.828156] ehci_hcd 0000:00:03.2: irq 23, io mem 0xe2004000 [ 1.834823] libata version 3.00 loaded. [ 1.840046] ehci_hcd 0000:00:03.2: USB 2.0 started, EHCI 1.00 [ 1.840126] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 1.840157] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.840191] usb usb1: Product: EHCI Host Controller [ 1.840221] usb usb1: Manufacturer: Linux 2.6.32-5-686 ehci_hcd [ 1.840251] usb usb1: SerialNumber: 0000:00:03.2 [ 1.840408] usb usb1: configuration #1 chosen from 1 choice [ 1.840473] hub 1-0:1.0: USB hub found [ 1.840512] hub 1-0:1.0: 6 ports detected [ 1.842024] pata_sis 0000:00:02.5: version 0.5.2 [ 1.842051] pata_sis 0000:00:02.5: PCI INT A -> GSI 16 (level, low) -> IRQ 16 [ 1.853698] thermal LNXTHERM:01: registered as thermal_zone0 [ 1.853743] ACPI: Thermal Zone [THRM] (58 C) [ 1.854095] scsi0 : pata_sis [ 1.855750] scsi1 : pata_sis [ 1.856030] ata1: PATA max UDMA/133 cmd 0x1f0 ctl 0x3f6 bmdma 0x2000 irq 14 [ 1.856062] ata2: PATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0x2008 irq 15 [ 2.443778] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.443825] ohci_hcd 0000:00:03.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20 [ 2.443850] ohci_hcd 0000:00:03.0: OHCI Host Controller [ 2.443868] ohci_hcd 0000:00:03.0: new USB bus registered, assigned bus number 2 [ 2.443901] ohci_hcd 0000:00:03.0: irq 20, io mem 0xe2002000 [ 2.446858] Console: switching to colour frame buffer device 128x48 [ 2.449960] sisfb: 2D acceleration is enabled, y-panning enabled (auto-max) [ 2.449963] fb0: SiS 760 frame buffer device version 1.8.9 [ 2.449965] sisfb: Copyright (C) 2001-2005 Thomas Winischhofer [ 2.456176] ata1.00: HPA unlocked: 268435455 -> 312581808, native 312581808 [ 2.456236] ata1.00: ATA-6: ST9160821A, 3.ALD, max UDMA/100 [ 2.456276] ata1.00: 312581808 sectors, multi 16: LBA48 [ 2.472313] ata1.00: configured for UDMA/100 [ 2.472474] scsi 0:0:0:0: Direct-Access ATA ST9160821A 3.AL PQ: 0 ANSI: 5 [ 2.496049] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001 [ 2.496104] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 2.496151] usb usb2: Product: OHCI Host Controller [ 2.496185] usb usb2: Manufacturer: Linux 2.6.32-5-686 ohci_hcd [ 2.496225] usb usb2: SerialNumber: 0000:00:03.0 [ 2.496369] usb usb2: configuration #1 chosen from 1 choice [ 2.496441] hub 2-0:1.0: USB hub found [ 2.496482] hub 2-0:1.0: 3 ports detected [ 2.496590] ohci_hcd 0000:00:03.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21 [ 2.496664] ohci_hcd 0000:00:03.1: OHCI Host Controller [ 2.496781] ohci_hcd 0000:00:03.1: new USB bus registered, assigned bus number 3 [ 2.496919] ohci_hcd 0000:00:03.1: irq 21, io mem 0xe2003000 [ 2.552037] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001 [ 2.552154] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 2.552260] usb usb3: Product: OHCI Host Controller [ 2.552353] usb usb3: Manufacturer: Linux 2.6.32-5-686 ohci_hcd [ 2.552452] usb usb3: SerialNumber: 0000:00:03.1 [ 2.552763] usb usb3: configuration #1 chosen from 1 choice [ 2.552955] hub 3-0:1.0: USB hub found [ 2.553060] hub 3-0:1.0: 3 ports detected [ 2.636217] ata2.00: ATAPI: Slimtype DVDRW SOSW-833S, VRS2, max UDMA/33 [ 2.652634] ata2.00: configured for UDMA/33 [ 2.653637] scsi 1:0:0:0: CD-ROM Slimtype DVDRW SOSW-833S VRS2 PQ: 0 ANSI: 5 [ 2.672391] b43-pci-bridge 0000:00:0b.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17 [ 2.687256] sd 0:0:0:0: [sda] 312581808 512-byte logical blocks: (160 GB/149 GiB) [ 2.687436] sd 0:0:0:0: [sda] Write Protect is off [ 2.687531] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 2.687553] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 2.687823] sda: [ 2.688774] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 2.689055] scsi 1:0:0:0: Attached scsi generic sg1 type 5 [ 2.695426] sda1 sda2 sda3 sda4 [ 2.708016] usb 1-4: new high speed USB device using ehci_hcd and address 3 [ 2.725488] sd 0:0:0:0: [sda] Attached SCSI disk [ 2.732116] ssb: Sonics Silicon Backplane found on PCI device 0000:00:0b.0 [ 2.840892] usb 1-4: New USB device found, idVendor=152d, idProduct=2338 [ 2.841023] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=5 [ 2.841130] usb 1-4: Product: USB to ATA/ATAPI bridge [ 2.841223] usb 1-4: Manufacturer: JMicron [ 2.841311] usb 1-4: SerialNumber: 000000000000 [ 2.841534] usb 1-4: configuration #1 chosen from 1 choice [ 2.853634] Initializing USB Mass Storage driver... [ 2.854053] scsi2 : SCSI emulation for USB Mass Storage devices [ 2.854636] usbcore: registered new interface driver usb-storage [ 2.854783] USB Mass Storage support registered. [ 2.856747] usb-storage: device found at 3 [ 2.856751] usb-storage: waiting for device to settle before scanning [ 3.016025] usb 2-2: new low speed USB device using ohci_hcd and address 2 [ 3.239859] usb 2-2: New USB device found, idVendor=1a2c, idProduct=0002 [ 3.239987] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 3.240097] usb 2-2: Product: USB Keykoard [ 3.240186] usb 2-2: Manufacturer: USB [ 3.240411] usb 2-2: configuration #1 chosen from 1 choice [ 3.260732] usbcore: registered new interface driver hiddev [ 3.267520] input: USB USB Keykoard as /devices/pci0000:00/0000:00:03.0/usb2/2-2/2-2:1.0/input/input1 [ 3.268050] generic-usb 0003:1A2C:0002.0001: input,hidraw0: USB HID v1.10 Keyboard [USB USB Keykoard] on usb-0000:00:03.0-2/input0 [ 7.856295] usb-storage: device scan complete [ 9.851963] scsi 2:0:0:0: Direct-Access ST950032 5AS PQ: 0 ANSI: 2 CCS [ 9.853597] sd 2:0:0:0: Attached scsi generic sg2 type 0 [ 12.560322] sd 2:0:0:0: [sdb] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 12.561190] sd 2:0:0:0: [sdb] Write Protect is off [ 12.561288] sd 2:0:0:0: [sdb] Mode Sense: 28 00 00 00 [ 12.561291] sd 2:0:0:0: [sdb] Assuming drive cache: write through [ 12.562814] sd 2:0:0:0: [sdb] Assuming drive cache: write through [ 12.562936] sdb: sdb1 [ 12.700712] sd 2:0:0:0: [sdb] Assuming drive cache: write through [ 12.700841] sd 2:0:0:0: [sdb] Attached SCSI disk [ 13.272670] /tmp/buildd/linux-2.6-2.6.32/debian/build/source_i386_none/drivers/hid/usbhid/hid-core.c: usb_submit_urb(ctrl) failed [ 13.272887] generic-usb 0003:1A2C:0002.0002: timeout initializing reports [ 13.273146] input: USB USB Keykoard as /devices/pci0000:00/0000:00:03.0/usb2/2-2/2-2:1.1/input/input2 [ 13.273346] generic-usb 0003:1A2C:0002.0002: input,hidraw1: USB HID v1.10 Device [USB USB Keykoard] on usb-0000:00:03.0-2/input1 [ 13.273567] usbcore: registered new interface driver usbhid [ 13.273666] usbhid: v2.6:USB HID core driver [ 13.506385] PM: Starting manual resume from disk [ 13.506508] PM: Resume from partition 8:4 [ 13.506510] PM: Checking hibernation image. [ 13.508419] PM: Error -22 checking image file [ 13.508421] PM: Resume from disk failed. [ 13.721670] EXT4-fs (sda2): mounted filesystem with ordered data mode [ 15.294222] <30>udevd[266]: starting version 172 [ 15.666363] input: Lid Switch as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0D:00/input/input3 [ 15.669244] ACPI: Lid Switch [LID] [ 15.669462] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input4 [ 15.669580] ACPI: Power Button [PWRB] [ 15.669743] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input5 [ 15.669858] ACPI: Sleep Button [SLPB] [ 15.670010] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input6 [ 15.670118] ACPI: Power Button [PWRF] [ 15.767070] ACPI: Battery Slot [BAT1] (battery present) [ 15.768531] ACPI: AC Adapter [ACAD] (on-line) [ 15.801205] Marking TSC unstable due to TSC halts in idle [ 15.801476] processor LNXCPU:00: registered as cooling_device0 [ 15.809072] Switching to clocksource acpi_pm [ 15.963087] yenta_cardbus 0000:00:06.0: CardBus bridge found [1025:0083] [ 15.963228] yenta_cardbus 0000:00:06.0: Using CSCINT to route CSC interrupts to PCI [ 15.963337] yenta_cardbus 0000:00:06.0: Routing CardBus interrupts to PCI [ 15.963442] yenta_cardbus 0000:00:06.0: TI: mfunc 0x00521d22, devctl 0x64 [ 16.056224] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 16.087536] ACPI: WMI: Mapper loaded [ 16.193069] yenta_cardbus 0000:00:06.0: ISA IRQ mask 0x06f8, PCI irq 19 [ 16.193202] yenta_cardbus 0000:00:06.0: Socket status: 30000006 [ 16.247795] input: PC Speaker as /devices/platform/pcspkr/input/input7 [ 16.250634] acer-wmi: Acer Laptop ACPI-WMI Extras [ 16.250761] acer-wmi: No or unsupported WMI interface, unable to load [ 16.448490] sis96x_smbus 0000:00:02.1: SiS96x SMBus base address: 0x8100 [ 16.449426] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 16.563486] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x100-0x3af: clean. [ 16.565348] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x3e0-0x4ff: excluding 0x480-0x48f [ 16.566124] pcmcia_socket pcmcia_socket0: cs: IO port probe 0x820-0x8ff: clean. [ 16.566790] pcmcia_socket pcmcia_socket0: cs: IO port probe 0xc00-0xcf7: clean. [ 16.567558] pcmcia_socket pcmcia_socket0: cs: IO port probe 0xa00-0xaff: clean. [ 16.655309] Synaptics Touchpad, model: 1, fw: 6.2, id: 0x12a0b1, caps: 0xa04713/0x204000/0x0 [ 16.690726] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input8 [ 17.958951] Adding 1012084k swap on /dev/sda4. Priority:-1 extents:1 across:1012084k [ 18.447701] loop: module loaded [ 20.312617] kjournald starting. Commit interval 5 seconds [ 20.312884] EXT3 FS on sda1, internal journal [ 20.312985] EXT3-fs: mounted filesystem with ordered data mode. [ 20.508748] EXT4-fs (sda3): mounted filesystem with ordered data mode [ 22.648592] RPC: Registered udp transport module. [ 22.648710] RPC: Registered tcp transport module. [ 22.648801] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 22.716383] Slow work thread pool: Starting up [ 22.716559] Slow work thread pool: Ready [ 22.716738] FS-Cache: Loaded [ 22.830414] FS-Cache: Netfs 'nfs' registered for caching [ 22.872615] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 23.044914] eth0: Media Link On 100mbps full-duplex [ 30.438147] ip_tables: (C) 2000-2006 Netfilter Core Team [ 30.882421] powernow-k8: Found 1 Mobile AMD Sempron(tm) Processor 3000+ processors (1 cpu cores) (version 2.20.00) [ 30.882642] powernow-k8: 0 : fid 0xa (1800 MHz), vid 0xa [ 30.882769] powernow-k8: 1 : fid 0x8 (1600 MHz), vid 0xc [ 30.882894] powernow-k8: 2 : fid 0x0 (800 MHz), vid 0x13 [ 30.933252] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory [ 30.948879] NFSD: starting 90-second grace period [ 32.020370] eth0: no IPv6 routers present ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: System freezes with high network activity 2011-12-02 20:54 ` Jose Luis Salas @ 2011-12-02 22:28 ` john stultz 2011-12-03 22:04 ` Jose Luis Salas 0 siblings, 1 reply; 7+ messages in thread From: john stultz @ 2011-12-02 22:28 UTC (permalink / raw) To: Jose Luis Salas; +Cc: linux-kernel, Thomas Gleixner, Jonathan Nieder On Fri, 2011-12-02 at 21:54 +0100, Jose Luis Salas wrote: > Hi, > > attached is the dmesg without the clocksource option. Thanks. After your done testing nohz=off, could you also send /proc/timer_list output from the system with no clocksource option, and no nohz options? > the nohz=off *seems* to avoid the problem, I'm testing with NFS and Iperf now. Sounds good. Let us know how the testing goes. thanks -john ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: System freezes with high network activity 2011-12-02 22:28 ` john stultz @ 2011-12-03 22:04 ` Jose Luis Salas 2012-03-17 1:41 ` John Stultz 0 siblings, 1 reply; 7+ messages in thread From: Jose Luis Salas @ 2011-12-03 22:04 UTC (permalink / raw) To: john stultz; +Cc: linux-kernel, Thomas Gleixner, Jonathan Nieder [-- Attachment #1: Type: text/plain, Size: 710 bytes --] Hi, attached is the output of the timer_list. With the nohz option the system is stable too. Other symptom of the problem is network drops performance to 50% ( 50 Mbps ). Thanks again. On Fri, Dec 2, 2011 at 11:28 PM, john stultz <johnstul@us.ibm.com> wrote: > On Fri, 2011-12-02 at 21:54 +0100, Jose Luis Salas wrote: >> Hi, >> >> attached is the dmesg without the clocksource option. > > Thanks. After your done testing nohz=off, could you also > send /proc/timer_list output from the system with no clocksource option, > and no nohz options? > > >> the nohz=off *seems* to avoid the problem, I'm testing with NFS and Iperf now. > > Sounds good. Let us know how the testing goes. > > thanks > -john > > [-- Attachment #2: timer_list.txt --] [-- Type: text/plain, Size: 8315 bytes --] root@tomberi:~# cat /proc/timer_list Timer List Version: v0.5 HRTIMER_MAX_CLOCK_BASES: 2 now at 223634459785 nsecs cpu: 0 clock 0: .base: c20038f8 .index: 0 .resolution: 1 nsecs .get_time: ktime_get_real .offset: 1322949270758217542 nsecs active timers: clock 1: .base: c2003924 .index: 1 .resolution: 1 nsecs .get_time: ktime_get .offset: 0 nsecs active timers: #0: <c2003988>, tick_sched_timer, S:01, hrtimer_start_range_ns, swapper/0 # expires at 223636000000-223636000000 nsecs [in 1540215 to 1540215 nsecs] #1: <f5dfff44>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, varnishd/1718 # expires at 223646022402-223646072402 nsecs [in 11562617 to 11612617 nsecs] #2: <f5dc9f44>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, varnishncsa/1700 # expires at 223667196113-223667246113 nsecs [in 32736328 to 32786328 nsecs] #3: <f5e87b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, munin-node/1795 # expires at 223705422995-223707422992 nsecs [in 70963210 to 72963207 nsecs] #4: <f5e03b88>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, varnishd/1720 # expires at 223708346664-223709346662 nsecs [in 73886879 to 74886877 nsecs] #5: <f67cfb88>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, php5-fpm/1181 # expires at 223733109437-223733239435 nsecs [in 98649652 to 98779650 nsecs] #6: <f5df5f44>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, varnishd/1713 # expires at 224439802836-224439852836 nsecs [in 805343051 to 805393051 nsecs] #7: <f5df9f44>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, varnishd/1715 # expires at 224439818760-224439868760 nsecs [in 805358975 to 805408975 nsecs] #8: <f5dfdf44>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, varnishd/1717 # expires at 224439829376-224439879376 nsecs [in 805369591 to 805419591 nsecs] #9: <f6575b88>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, apache2/1225 # expires at 224481716600-224484716598 nsecs [in 847256815 to 850256813 nsecs] #10: <f6795b88>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, rrdcached/1127 # expires at 224531231434-224532231432 nsecs [in 896771649 to 897771647 nsecs] #11: <f64b3b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, apache2/1217 # expires at 224531270266-224532270264 nsecs [in 896810481 to 897810479 nsecs] #12: <f5d31b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1453 # expires at 224535982121-224536982118 nsecs [in 901522336 to 902522333 nsecs] #13: <f5d29b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1444 # expires at 224536149464-224537149453 nsecs [in 901689679 to 902689668 nsecs] #14: <f5d35b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1455 # expires at 224536224056-224537224054 nsecs [in 901764271 to 902764269 nsecs] #15: <f5d27b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1445 # expires at 224536320160-224537320158 nsecs [in 901860375 to 902860373 nsecs] #16: <f5d1fb10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1452 # expires at 224536482474-224537482472 nsecs [in 902022689 to 903022687 nsecs] #17: <f5d23b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1442 # expires at 224536541421-224537541419 nsecs [in 902081636 to 903081634 nsecs] #18: <f642db10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1450 # expires at 224536581371-224537581369 nsecs [in 902121586 to 903121584 nsecs] #19: <f5d37b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1456 # expires at 224536669093-224537669091 nsecs [in 902209308 to 903209306 nsecs] #20: <f5d33b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1454 # expires at 224537194867-224538194865 nsecs [in 902735082 to 903735080 nsecs] #21: <f5d2fb10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1448 # expires at 224537318628-224538318626 nsecs [in 902858843 to 903858841 nsecs] #22: <f5d1db10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1451 # expires at 224537436243-224538436241 nsecs [in 902976458 to 903976456 nsecs] #23: <f5d39b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1457 # expires at 224537494072-224538494070 nsecs [in 903034287 to 904034285 nsecs] #24: <f6f5bf44>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, powernowd/1486 # expires at 224563283840-224563333840 nsecs [in 928824055 to 928874055 nsecs] #25: <f5d06030>, posix_timer_fn, S:01, hrtimer_start_range_ns, ntpd/1435 # expires at 224613224725-224613224725 nsecs [in 978764940 to 978764940 nsecs] #26: <f6c2bb10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, init/1 # expires at 225879528033-225884528031 nsecs [in 2245068248 to 2250068246 nsecs] #27: <f5dd9b88>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, varnishd/1711 # expires at 226515274520-226518273519 nsecs [in 2880814735 to 2883813734 nsecs] #28: <f66fdb88>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, rpcbind/894 # expires at 227068145926-227098145923 nsecs [in 3433686141 to 3463686138 nsecs] #29: <f5d45b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, nmbd/1615 # expires at 230422309607-230432309604 nsecs [in 6787849822 to 6797849819 nsecs] #30: <f67efb10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, fail2ban-server/1411 # expires at 243848183633-243878183630 nsecs [in 20213723848 to 20243723845 nsecs] #31: <f5ce1f44>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, cron/1599 # expires at 270838974289-270839024289 nsecs [in 47204514504 to 47204564504 nsecs] #32: <f5d6bb10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, smbd/1619 # expires at 276733246521-276793234139 nsecs [in 53098786736 to 53158774354 nsecs] #33: <f6fdeb84>, it_real_fn, S:01, hrtimer_start, qmgr/1598 # expires at 368656511055-368656511055 nsecs [in 145022051270 to 145022051270 nsecs] #34: <f5dfbf44>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, varnishd/1716 # expires at 403407088880-403407138880 nsecs [in 179772629095 to 179772679095 nsecs] #35: <f6f8e944>, it_real_fn, S:01, hrtimer_start, master/1589 # expires at 548656401260-548656401260 nsecs [in 325021941475 to 325021941475 nsecs] #36: <f5dc5f44>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, smartd/1659 # expires at 1837849481974-1837849531974 nsecs [in 1614215022189 to 1614215072189 nsecs] #37: <f6791dd4>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, rrdcached/1142 # expires at 3625809309214-3625809359214 nsecs [in 3402174849429 to 3402174899429 nsecs] #38: <f6f8e704>, it_real_fn, S:01, hrtimer_start, pickup/1597 # expires at 6215656362427-6215656362427 nsecs [in 5992021902642 to 5992021902642 nsecs] #39: <f65f1b10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, smbd/1630 # expires at 10035509749904-10035609749904 nsecs [in 9811875290119 to 9811975290119 nsecs] #40: <f64efb10>, hrtimer_wakeup, S:01, hrtimer_start_range_ns, rsyslogd/1148 # expires at 86426401619630-86426501619630 nsecs [in 86202767159845 to 86202867159845 nsecs] .expires_next : 223636000000 nsecs .hres_active : 1 .nr_events : 23600 .nr_retries : 2 .nr_hangs : 0 .max_hang_time : 0 nsecs .nohz_mode : 2 .idle_tick : 223624000000 nsecs .tick_stopped : 0 .idle_jiffies : 4294948201 .idle_calls : 207772 .idle_sleeps : 70046 .idle_entrytime : 223627956888 nsecs .idle_waketime : 223624004081 nsecs .idle_exittime : 223624050457 nsecs .idle_sleeptime : 189861828990 nsecs .last_jiffies : 4294948202 .next_jiffies : 4294948203 .idle_expires : 223628000000 nsecs jiffies: 4294948204 Tick Device: mode: 1 Broadcast device Clock Event Device: pit max_delta_ns: 27461866 min_delta_ns: 12571 mult: 5124677 shift: 32 mode: 3 next_event: 9223372036854775807 nsecs set_next_event: pit_next_event set_mode: init_pit_timer event_handler: tick_handle_oneshot_broadcast tick_broadcast_mask: 00000001 tick_broadcast_oneshot_mask: 00000000 Tick Device: mode: 1 Per CPU device: 0 Clock Event Device: lapic max_delta_ns: 671068775 min_delta_ns: 1199 mult: 53688674 shift: 32 mode: 3 next_event: 223636000000 nsecs set_next_event: lapic_next_event set_mode: lapic_timer_setup event_handler: hrtimer_interrupt ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: System freezes with high network activity 2011-12-03 22:04 ` Jose Luis Salas @ 2012-03-17 1:41 ` John Stultz 2012-03-20 20:02 ` Jose Luis Salas 0 siblings, 1 reply; 7+ messages in thread From: John Stultz @ 2012-03-17 1:41 UTC (permalink / raw) To: Jose Luis Salas; +Cc: linux-kernel, Thomas Gleixner, Jonathan Nieder On 12/03/2011 02:04 PM, Jose Luis Salas wrote: > Hi, > > attached is the output of the timer_list. > > With the nohz option the system is stable too. > > Other symptom of the problem is network drops performance to 50% ( 50 Mbps ). > Hey Jose, Just following up on this old email. Looking at the timer_list.txt that you sent, I'm not seeing anything that sticks out as problematic. Are you still seeing issues with recent kernels (3.1, 3.2) ? Is nohz still working for you? I suspect the problem is that the lapic on your machine goes out to lunch after longish idle times w/ nohz. That's why the key-press or network traffic wakes the system back up. Does booting with the following patch (without nohz or any clocksource= boot args) fix the issue? If it does, can you increase the time returned in the patch from 20ms by powers of ten until it gets to 2 seconds or you see the problem? If the patch below doesn't help, can you drop the value down to 1ms and let me know if that affects anything? thanks -john diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 95bebaa..8fd2bfa 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -546,6 +546,9 @@ u64 timekeeping_max_deferment(void) { unsigned long seq; u64 ret; + + return 20000000ULL; /* 20ms */ + do { seq = read_seqbegin(&timekeeper.lock); ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: System freezes with high network activity 2012-03-17 1:41 ` John Stultz @ 2012-03-20 20:02 ` Jose Luis Salas 0 siblings, 0 replies; 7+ messages in thread From: Jose Luis Salas @ 2012-03-20 20:02 UTC (permalink / raw) To: linux-kernel Hello John, I'll wait for 3.3 Debian kernel package to test it and send you the results. Yes, my system is working properly with the nohz option in 39 days. I'll try the patch asap. Thanks for your great work. On Sat, Mar 17, 2012 at 2:41 AM, John Stultz <johnstul@us.ibm.com> wrote: > > On 12/03/2011 02:04 PM, Jose Luis Salas wrote: >> >> Hi, >> >> attached is the output of the timer_list. >> >> With the nohz option the system is stable too. >> >> Other symptom of the problem is network drops performance to 50% ( 50 >> Mbps ). >> > > Hey Jose, > Just following up on this old email. Looking at the timer_list.txt that > you sent, I'm not seeing anything that sticks out as problematic. Are you > still seeing issues with recent kernels (3.1, 3.2) ? Is nohz still > working for you? > > I suspect the problem is that the lapic on your machine goes out to lunch > after longish idle times w/ nohz. That's why the key-press or network > traffic wakes the system back up. > > Does booting with the following patch (without nohz or any clocksource= > boot args) fix the issue? > > If it does, can you increase the time returned in the patch from 20ms by > powers of ten until it gets to 2 seconds or you see the problem? If the > patch below doesn't help, can you drop the value down to 1ms and let me know > if that affects anything? > > thanks > -john > > diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c > index 95bebaa..8fd2bfa 100644 > --- a/kernel/time/timekeeping.c > +++ b/kernel/time/timekeeping.c > @@ -546,6 +546,9 @@ u64 timekeeping_max_deferment(void) > { > unsigned long seq; > u64 ret; > + > + return 20000000ULL; /* 20ms */ > + > do { > seq = read_seqbegin(&timekeeper.lock); > > ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2012-03-20 20:02 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-12-02 16:44 System freezes with high network activity Jose Luis Salas 2011-12-02 19:43 ` john stultz 2011-12-02 20:54 ` Jose Luis Salas 2011-12-02 22:28 ` john stultz 2011-12-03 22:04 ` Jose Luis Salas 2012-03-17 1:41 ` John Stultz 2012-03-20 20:02 ` Jose Luis Salas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox