2.6.13-rc6-rt3

2.6.13-rc6-rt3

[Ingo Molnar]

i have released the 2.6.13-rc6-rt3 tree, which can be downloaded from 
the usual place:

  http://redhat.com/~mingo/realtime-preempt/

Changes since 2.6.13-rc6-rt2:

 - reverted an incorrect tasklist_lock -> lock_task() conversion that 
   slipped in via an earlier hack in the HRT code. The issue it tried to 
   solve is mooted by the RCU-tasklist-lock code.

 - introduce mutex_chprio() as the mechanism for High Resolution timers 
   to dynamically change the priority of the HRT softirq thread. [This
   fixes a latency/priority bug in where the dynamic prio would override 
   the inherited priority, and when the PI mechanism restores priority 
   it would override the HRT priority.]

 - now that HR-timer dynamic priorities have the proper framework, 
   removed the HIGH_RES_TIMERS_DYN_PRIO config option and made the 
   feature unconditional.

to build a 2.6.13-rc6-rt3 tree, the following patches should be applied:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.12.tar.bz2
   http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.13-rc6.bz2
   http://redhat.com/~mingo/realtime-preempt/patch-2.6.13-rc6-rt3

	Ingo

Re: 2.6.13-rc6-rt5

[Steven Rostedt]

I got the following on compile:

cd ~/work/kernels/svn/linux_realtime_ernie/
make 
  CHK     include/linux/version.h
make[1]: `arch/i386/kernel/asm-offsets.s' is up to date.
  CHK     include/asm-i386/asm_offsets.h
  CHK     include/linux/compile.h
  CHK     usr/initramfs_list
  CC      kernel/latency.o
kernel/latency.c: In function 'check_critical_timing':
kernel/latency.c:1298: warning: 'flags' may be used uninitialized in this function
kernel/latency.c: In function '__start_critical_timing':
kernel/latency.c:1444: error: too few arguments to function '____trace'
kernel/latency.c: In function '__stop_critical_timing':
kernel/latency.c:1462: error: too few arguments to function '____trace'
make[1]: *** [kernel/latency.o] Error 1
make: *** [kernel] Error 2


Here's a patch.  I'm assuming this is what you want with flags.  I
haven't looked too deep into it ____trace. (I only look down a few '_'
not 4 of them :-)

-- Steve

Signed-off-by: Steven Rostedt

Index: linux_realtime_ernie/kernel/latency.c
===================================================================
--- linux_realtime_ernie/kernel/latency.c	(revision 293)
+++ linux_realtime_ernie/kernel/latency.c	(working copy)
@@ -1295,7 +1295,7 @@
 {
 	unsigned long latency, t0, t1;
 	cycles_t T0, T1, T2, delta;
-	unsigned long flags;
+	unsigned long flags = 0;
 
 	if (trace_user_triggered)
 		return;
@@ -1441,7 +1441,7 @@
 	_trace_cmdline(cpu, tr);
 
 	raw_local_save_flags(flags);
-	____trace(cpu, TRACE_FN, tr, eip, parent_eip, 0, 0, 0);
+	____trace(cpu, TRACE_FN, tr, eip, parent_eip, 0, 0, 0, flags);
 
 	atomic_dec(&tr->disabled);
 }
@@ -1459,7 +1459,7 @@
 
 	atomic_inc(&tr->disabled);
 	raw_local_save_flags(flags);
-	____trace(cpu, TRACE_FN, tr, eip, parent_eip, 0, 0, 0);
+	____trace(cpu, TRACE_FN, tr, eip, parent_eip, 0, 0, 0, flags);
 	check_critical_timing(cpu, tr, eip);
 	tr->critical_start = 0;
 	atomic_dec(&tr->disabled);

Re: 2.6.13-rc6-rt5

[Steven Rostedt]

Ouch, what was I thinking for that initializing flags to zero:

Sorry, lets try that again:

Signed-off-by: Steven Rostedt


Index: linux_realtime_ernie/kernel/latency.c
===================================================================
--- linux_realtime_ernie/kernel/latency.c	(revision 293)
+++ linux_realtime_ernie/kernel/latency.c	(working copy)
@@ -1307,12 +1307,13 @@
 	T1 = cycles();
 	delta = T1-T0;
 
+	raw_local_save_flags(flags);
+
 #ifndef CONFIG_CRITICAL_LATENCY_HIST
 	if (!report_latency(delta))
 		goto out;
 #endif
 
-	raw_local_save_flags(flags);
 	____trace(cpu, TRACE_FN, tr, CALLER_ADDR0, parent_eip, 0, 0, 0, flags);
 	/*
 	 * Update the timestamp, because the trace entry above
@@ -1441,7 +1442,7 @@
 	_trace_cmdline(cpu, tr);
 
 	raw_local_save_flags(flags);
-	____trace(cpu, TRACE_FN, tr, eip, parent_eip, 0, 0, 0);
+	____trace(cpu, TRACE_FN, tr, eip, parent_eip, 0, 0, 0, flags);
 
 	atomic_dec(&tr->disabled);
 }
@@ -1459,7 +1460,7 @@
 
 	atomic_inc(&tr->disabled);
 	raw_local_save_flags(flags);
-	____trace(cpu, TRACE_FN, tr, eip, parent_eip, 0, 0, 0);
+	____trace(cpu, TRACE_FN, tr, eip, parent_eip, 0, 0, 0, flags);
 	check_critical_timing(cpu, tr, eip);
 	tr->critical_start = 0;
 	atomic_dec(&tr->disabled);

Re: 2.6.13-rc6-rt5

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 250 bytes --]

OK, I compiled it now. Tried to boot, and it rebooted on me before
showing any output :-(

So I'll start debugging it, but just incase I'm not looking for
something that is already found, is there anything wrong with the
following config?

-- Steve


[-- Attachment #2: config_ernie --]
[-- Type: text/plain, Size: 5699 bytes --]

CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSCTL=y
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_EMBEDDED=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_OBSOLETE_MODPARM=y
CONFIG_KMOD=y
CONFIG_X86_PC=y
CONFIG_MPENTIUMIII=y
CONFIG_X86_GENERIC=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_HIGH_RES_TIMER_TSC=y
CONFIG_HIGH_RES_RESOLUTION=1000
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_STATS=y
CONFIG_RCU_TORTURE_TEST=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_ASM_SEMAPHORES=y
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_IOAPIC_FAST=y
CONFIG_X86_TSC=y
CONFIG_NOHIGHMEM=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_MTRR=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_SECCOMP=y
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_PHYSICAL_START=0x100000
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_MISC=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_FIB_HASH=y
CONFIG_IP_TCPDIAG=y
CONFIG_TCP_CONG_BIC=y
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_PARPORT=y
CONFIG_PNP=y
CONFIG_ISAPNP=y
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_INITRAMFS_SOURCE=""
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_CMD640=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_RZ1000=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
CONFIG_SCSI=y
CONFIG_SCSI_PROC_FS=y
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_QLA2XXX=y
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_NET_PCI=y
CONFIG_E100=y
CONFIG_NETCONSOLE=m
CONFIG_NETPOLL=y
CONFIG_NETPOLL_RX=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_INPORT=m
CONFIG_MOUSE_LOGIBM=m
CONFIG_MOUSE_PC110PAD=m
CONFIG_MOUSE_VSXXXAA=m
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
CONFIG_SERIO_LIBPS2=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_RTC=y
CONFIG_RTC_HISTOGRAM=y
CONFIG_BLOCKER=y
CONFIG_AGP=y
CONFIG_AGP_ALI=y
CONFIG_AGP_AMD=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
CONFIG_DRM=y
CONFIG_DRM_TDFX=y
CONFIG_DRM_RADEON=y
CONFIG_HWMON=y
CONFIG_FB=y
CONFIG_VIDEO_SELECT=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FONTS=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB=y
CONFIG_USB_STORAGE=y
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
CONFIG_USB_HIDDEV=y
CONFIG_USB_MON=y
CONFIG_EXT2_FS=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_JBD=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
CONFIG_INOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_AUTOFS4_FS=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_PROC_FS=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_NFS_FS=y
CONFIG_NFSD=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=y
CONFIG_EXPORTFS=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_LOG_BUF_SHIFT=14
CONFIG_DETECT_SOFTLOCKUP=y
CONFIG_SCHEDSTATS=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_IRQ_FLAGS=y
CONFIG_WAKEUP_TIMING=y
CONFIG_PREEMPT_TRACE=y
CONFIG_CRITICAL_PREEMPT_TIMING=y
CONFIG_CRITICAL_IRQSOFF_TIMING=y
CONFIG_CRITICAL_TIMING=y
CONFIG_LATENCY_TIMING=y
CONFIG_LATENCY_TRACE=y
CONFIG_MCOUNT=y
CONFIG_RT_DEADLOCK_DETECT=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_FRAME_POINTER=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_CRC32=y
CONFIG_ZLIB_INFLATE=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y

Re: 2.6.13-rc6-rt5

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> OK, I compiled it now. Tried to boot, and it rebooted on me before 
> showing any output :-(
> 
> So I'll start debugging it, but just incase I'm not looking for 
> something that is already found, is there anything wrong with the 
> following config?

please send non-stripped .configs if possible, if you want me to try and 
reproduce your problem.

	Ingo

Re: 2.6.13-rc6-rt5

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> OK, I compiled it now. Tried to boot, and it rebooted on me before
> showing any output :-(
> 
> So I'll start debugging it, but just incase I'm not looking for
> something that is already found, is there anything wrong with the
> following config?

your .config reboots here too. I suspect it's one of the latency tracing 
options that causes it. Perhaps some trace recursion?

	Ingo

Re: 2.6.13-rc6-rt5

[Ingo Molnar]

i removed some debug options from your .config, and that booted. Here's 
the diff between the good and bad .config's. Trying to narrow it down 
further.

	Ingo

--- .config.good00
+++ .config.bad00
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.13-rc6-rt6
-# Tue Aug 16 18:10:47 2005
+# Tue Aug 16 18:05:14 2005
 #
 CONFIG_X86=y
 CONFIG_MMU=y
@@ -156,7 +156,6 @@
 # CONFIG_MATH_EMULATION is not set
 CONFIG_MTRR=y
 CONFIG_HAVE_DEC_LOCK=y
-# CONFIG_REGPARM is not set
 CONFIG_SECCOMP=y
 # CONFIG_HZ_100 is not set
 # CONFIG_HZ_250 is not set
@@ -1239,23 +1238,28 @@
 CONFIG_LOG_BUF_SHIFT=14
 CONFIG_DETECT_SOFTLOCKUP=y
 CONFIG_SCHEDSTATS=y
-# CONFIG_DEBUG_SLAB is not set
-# CONFIG_DEBUG_PREEMPT is not set
-# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
-# CONFIG_WAKEUP_TIMING is not set
+CONFIG_DEBUG_SLAB=y
+CONFIG_DEBUG_PREEMPT=y
+CONFIG_DEBUG_IRQ_FLAGS=y
+CONFIG_WAKEUP_TIMING=y
+# CONFIG_WAKEUP_LATENCY_HIST is not set
+CONFIG_PREEMPT_TRACE=y
 # CONFIG_CRITICAL_PREEMPT_TIMING is not set
 # CONFIG_CRITICAL_IRQSOFF_TIMING is not set
-# CONFIG_RT_DEADLOCK_DETECT is not set
+CONFIG_LATENCY_TIMING=y
+CONFIG_LATENCY_TRACE=y
+CONFIG_MCOUNT=y
+CONFIG_RT_DEADLOCK_DETECT=y
 # CONFIG_DEBUG_RT_LOCKING_MODE is not set
 # CONFIG_DEBUG_KOBJECT is not set
 CONFIG_DEBUG_BUGVERBOSE=y
-# CONFIG_DEBUG_INFO is not set
+CONFIG_DEBUG_INFO=y
 # CONFIG_DEBUG_FS is not set
-# CONFIG_USE_FRAME_POINTER is not set
+CONFIG_FRAME_POINTER=y
 CONFIG_EARLY_PRINTK=y
-# CONFIG_DEBUG_STACKOVERFLOW is not set
+CONFIG_DEBUG_STACKOVERFLOW=y
 # CONFIG_KPROBES is not set
-# CONFIG_DEBUG_STACK_USAGE is not set
+CONFIG_DEBUG_STACK_USAGE=y
 # CONFIG_DEBUG_PAGEALLOC is not set
 # CONFIG_4KSTACKS is not set
 CONFIG_X86_FIND_SMP_CONFIG=y

Re: 2.6.13-rc6-rt5

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> i removed some debug options from your .config, and that booted. 
> Here's the diff between the good and bad .config's. Trying to narrow 
> it down further.

even narrower diff below. My guess would be on 
CONFIG_DEBUG_STACKOVERFLOW.

	Ingo

--- .config.good01	2005-08-16 18:19:05.410509952 +0200
+++ .config.bad00	2005-08-16 18:10:00.242388104 +0200
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.13-rc6-rt6
-# Tue Aug 16 18:13:48 2005
+# Tue Aug 16 18:05:14 2005
 #
 CONFIG_X86=y
 CONFIG_MMU=y
@@ -156,7 +156,6 @@
 # CONFIG_MATH_EMULATION is not set
 CONFIG_MTRR=y
 CONFIG_HAVE_DEC_LOCK=y
-# CONFIG_REGPARM is not set
 CONFIG_SECCOMP=y
 # CONFIG_HZ_100 is not set
 # CONFIG_HZ_250 is not set
@@ -1242,21 +1241,25 @@
 CONFIG_DEBUG_SLAB=y
 CONFIG_DEBUG_PREEMPT=y
 CONFIG_DEBUG_IRQ_FLAGS=y
-# CONFIG_WAKEUP_TIMING is not set
+CONFIG_WAKEUP_TIMING=y
+# CONFIG_WAKEUP_LATENCY_HIST is not set
 CONFIG_PREEMPT_TRACE=y
 # CONFIG_CRITICAL_PREEMPT_TIMING is not set
 # CONFIG_CRITICAL_IRQSOFF_TIMING is not set
+CONFIG_LATENCY_TIMING=y
+CONFIG_LATENCY_TRACE=y
+CONFIG_MCOUNT=y
 CONFIG_RT_DEADLOCK_DETECT=y
 # CONFIG_DEBUG_RT_LOCKING_MODE is not set
 # CONFIG_DEBUG_KOBJECT is not set
 CONFIG_DEBUG_BUGVERBOSE=y
-# CONFIG_DEBUG_INFO is not set
+CONFIG_DEBUG_INFO=y
 # CONFIG_DEBUG_FS is not set
-# CONFIG_USE_FRAME_POINTER is not set
+CONFIG_FRAME_POINTER=y
 CONFIG_EARLY_PRINTK=y
-# CONFIG_DEBUG_STACKOVERFLOW is not set
+CONFIG_DEBUG_STACKOVERFLOW=y
 # CONFIG_KPROBES is not set
-# CONFIG_DEBUG_STACK_USAGE is not set
+CONFIG_DEBUG_STACK_USAGE=y
 # CONFIG_DEBUG_PAGEALLOC is not set
 # CONFIG_4KSTACKS is not set
 CONFIG_X86_FIND_SMP_CONFIG=y

Re: 2.6.13-rc6-rt5

[Ingo Molnar]

it's the raw_local_irq_save() in ___trace() that causes trouble.

	Ingo

2.6.13-rc6-rt6

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> it's the raw_local_irq_save() in ___trace() that causes trouble.

ok, i've uploaded 2.6.13-rc6-rt6, which should fix this. (i've pushed 
the IRQ tracing into the raw_local_*() primitives, and kept the 
__raw_local_*() primitives clean, and ___trace() is using them now)

Does it boot for you now?

-rt6 also includes the USB fixes from Alan Stern, and an x64 fix from 
Dave Jiang. (albeit x64 likely doesnt work yet)

	Ingo

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Tue, 2005-08-16 at 19:08 +0200, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > it's the raw_local_irq_save() in ___trace() that causes trouble.
> 
> ok, i've uploaded 2.6.13-rc6-rt6, which should fix this. (i've pushed 
> the IRQ tracing into the raw_local_*() primitives, and kept the 
> __raw_local_*() primitives clean, and ___trace() is using them now)
> 
> Does it boot for you now?

Ingo,

I just compiled it and got a bunch of uninitialized variable warnings.
Are you sure you want this?  Here's the problem.


In something like _trace_cmdline we have:

static void notrace _trace_cmdline(int cpu, struct cpu_trace *tr)
{
	unsigned long flags;

	raw_local_save_flags(flags);
	____trace(cpu, TRACE_CMDLINE, tr, 0, 0, 0, 0, 0, flags);
}


Now the raw_local_save_flags is defined as:

do { typecheck(unsigned long,flags); \
    if (raw_irqs_disabled_flags(flags)) \
        trace_irqs_off(); else trace_irqs_on(); \
        __raw_local_save_flags(flags); } while (0)


The test of raw_irqs_disabled_flags is checking an uninitialized
variable.  I haven't tried to run it yet since this just doesn't look
good.

I changed it for now to the following, but it still desn't make sense to
me.  With a local_save_flags, which doesn't disable or restore the
interrupts, why bother with the trace at all?

-- Steve

Index: linux_realtime_ernie/include/linux/rt_irq.h
===================================================================
--- linux_realtime_ernie/include/linux/rt_irq.h (revision 294)
+++ linux_realtime_ernie/include/linux/rt_irq.h (working copy)
@@ -26,7 +26,7 @@
 # endif

 /* soft state does not follow the hard state */
-# define raw_local_save_flags(flags)   do { typecheck(unsigned long,flags); if (raw_irqs_disabled_flags(flags)) trace_irqs_off(); else trace_irqs_on(); __raw_local_save_flags(flags); } while (0)
+# define raw_local_save_flags(flags)   do { typecheck(unsigned long,flags); if (raw_irqs_disabled()) trace_irqs_off(); else trace_irqs_on(); __raw_local_save_flags(flags); } while (0)
 # define raw_local_irq_enable()                do { trace_irqs_on(); __raw_local_irq_enable(); } while (0)
 # define raw_local_irq_disable()       do { __raw_local_irq_disable(); trace_irqs_off(); } while (0)
 # define raw_local_irq_save(flags)     do { __raw_local_irq_save(flags); trace_irqs_off(); } while (0)

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Tue, 2005-08-16 at 13:50 -0400, Steven Rostedt wrote:

> I changed it for now to the following, but it still desn't make sense to
> me.  With a local_save_flags, which doesn't disable or restore the
> interrupts, why bother with the trace at all?

Or should the following patch really be applied? (it boots and compiles
nicely).

-- Steve

Index: linux_realtime_ernie/include/linux/rt_irq.h
===================================================================
--- linux_realtime_ernie/include/linux/rt_irq.h	(revision 294)
+++ linux_realtime_ernie/include/linux/rt_irq.h	(working copy)
@@ -26,7 +26,7 @@
 # endif
 
 /* soft state does not follow the hard state */
-# define raw_local_save_flags(flags)	do { typecheck(unsigned long,flags); if (raw_irqs_disabled_flags(flags)) trace_irqs_off(); else trace_irqs_on(); __raw_local_save_flags(flags); } while (0)
+# define raw_local_save_flags(flags)	do { typecheck(unsigned long,flags);  __raw_local_save_flags(flags); } while (0)
 # define raw_local_irq_enable()		do { trace_irqs_on(); __raw_local_irq_enable(); } while (0)
 # define raw_local_irq_disable()	do { __raw_local_irq_disable(); trace_irqs_off(); } while (0)
 # define raw_local_irq_save(flags)	do { __raw_local_irq_save(flags); trace_irqs_off(); } while (0)

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Tue, 2005-08-16 at 14:07 -0400, Steven Rostedt wrote:
> On Tue, 2005-08-16 at 13:50 -0400, Steven Rostedt wrote:
> 
> > I changed it for now to the following, but it still desn't make sense to
> > me.  With a local_save_flags, which doesn't disable or restore the
> > interrupts, why bother with the trace at all?
> 
> Or should the following patch really be applied? (it boots and compiles
> nicely).
> 

Since the change made raw_local_save_flags the same for both PREEMPT_RT
and !PREEMPT_RT, I moved it out of the #ifdef altogether.  The
__raw_local_save_flags already does the type checking (at least for
intel).

-- Steve

Index: linux_realtime_ernie/include/linux/rt_irq.h
===================================================================
--- linux_realtime_ernie/include/linux/rt_irq.h	(revision 294)
+++ linux_realtime_ernie/include/linux/rt_irq.h	(working copy)
@@ -26,7 +26,6 @@
 # endif
 
 /* soft state does not follow the hard state */
-# define raw_local_save_flags(flags)	do { typecheck(unsigned long,flags); if (raw_irqs_disabled_flags(flags)) trace_irqs_off(); else trace_irqs_on(); __raw_local_save_flags(flags); } while (0)
 # define raw_local_irq_enable()		do { trace_irqs_on(); __raw_local_irq_enable(); } while (0)
 # define raw_local_irq_disable()	do { __raw_local_irq_disable(); trace_irqs_off(); } while (0)
 # define raw_local_irq_save(flags)	do { __raw_local_irq_save(flags); trace_irqs_off(); } while (0)
@@ -35,7 +34,6 @@
 # define raw_safe_halt()		__raw_safe_halt()
 #else
 # define RAW_LOCAL_ILLEGAL_MASK		0UL
-# define raw_local_save_flags		__raw_local_save_flags
 # define raw_local_irq_enable		__raw_local_irq_enable
 # define raw_local_irq_disable		__raw_local_irq_disable
 # define raw_local_irq_save		__raw_local_irq_save
@@ -50,6 +48,7 @@
 # define irqs_disabled_flags		__raw_irqs_disabled_flags
 #endif
 
+#define raw_local_save_flags		__raw_local_save_flags
 #define raw_irqs_disabled		__raw_irqs_disabled
 #define raw_irqs_disabled_flags		__raw_irqs_disabled_flags

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 1179 bytes --]

Ingo,

FYI, I ran this on my laptop (Pentium4 HT) and it locked up shortly
after it started INIT.  I rebooted, and now it's up and running 
with no problems!?!  I'll reboot it a few more times to see if it will
lock up again.

Unfortunately, my laptop doesn't have any serial so I don't have much to
debug with it.  I'll set up netconsole, but this is not as reliable
under RT, or was this fixed. I remember before that it could call things
that would schedule and had some hooks to keep interrupts on.

Oh, I also did get the message when it started:

WARNING: kstopmachine/859 changed soft IRQ-flags.
 [<c0149e0b>] stop_machine+0x11b/0x160 (8)
 [<c0149e7e>] do_stop+0xe/0x70 (32)
 [<c0139c5a>] kthread+0xba/0xc0 (16)
 [<c0139ba0>] kthread+0x0/0xc0 (28)
 [<c0101385>] kernel_thread_helper+0x5/0x10 (16)
---------------------------
| preempt count: 00000000 ]
| 0-level deep critical section nesting:
----------------------------------------

------------------------------
| showing all locks held by: |  (kstopmachine/859 [f74dc840,   0]):
------------------------------

The kernel I used was rt6 with my latest patch to the
raw_local_save_flags.

Config attached.

-- Steve


[-- Attachment #2: config-bilbo.bz2 --]
[-- Type: application/x-bzip, Size: 8789 bytes --]

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 00:20 -0400, Steven Rostedt wrote:
> Ingo,
> 
> FYI, I ran this on my laptop (Pentium4 HT) and it locked up shortly
> after it started INIT.  I rebooted, and now it's up and running 
> with no problems!?!  I'll reboot it a few more times to see if it will
> lock up again.

With a small change it locks up all the time :-)

> 
> Unfortunately, my laptop doesn't have any serial so I don't have much to
> debug with it.  I'll set up netconsole, but this is not as reliable
> under RT, or was this fixed. I remember before that it could call things
> that would schedule and had some hooks to keep interrupts on.
> 
> Oh, I also did get the message when it started:
> 
> WARNING: kstopmachine/859 changed soft IRQ-flags.
>  [<c0149e0b>] stop_machine+0x11b/0x160 (8)
>  [<c0149e7e>] do_stop+0xe/0x70 (32)
>  [<c0139c5a>] kthread+0xba/0xc0 (16)
>  [<c0139ba0>] kthread+0x0/0xc0 (28)
>  [<c0101385>] kernel_thread_helper+0x5/0x10 (16)

I replaced this with raw_local_irq_disable and it really locks up
everytime now.  I need to look into how this is best done.

> The kernel I used was rt6 with my latest patch to the
> raw_local_save_flags.

I just got the NMI working, but since it is now 1:44am my time, I'm
going to bed.  I'll run it again tomorrow and hopefully the NMI will
show where and why it's locking up.

-- Steve

Re: 2.6.13-rc6-rt6

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> Since the change made raw_local_save_flags the same for both 
> PREEMPT_RT and !PREEMPT_RT, I moved it out of the #ifdef altogether.  
> The __raw_local_save_flags already does the type checking (at least 
> for intel).

ok, i've applied this one. Indeed it's pointless to trace the 
local_save_flags API.

	Ingo

Re: 2.6.13-rc6-rt6

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Wed, 2005-08-17 at 00:20 -0400, Steven Rostedt wrote:
> > Ingo,
> > 
> > FYI, I ran this on my laptop (Pentium4 HT) and it locked up shortly
> > after it started INIT.  I rebooted, and now it's up and running 
> > with no problems!?!  I'll reboot it a few more times to see if it will
> > lock up again.
> 
> With a small change it locks up all the time :-)

unfortunately the space of "patches that break the kernel" is infinitely 
larger (and infinitely easier to generate) than the space of "patches 
that improve the kernel" - so i'll skip your patch for now ;-)

> > Oh, I also did get the message when it started:
> > 
> > WARNING: kstopmachine/859 changed soft IRQ-flags.
> >  [<c0149e0b>] stop_machine+0x11b/0x160 (8)
> >  [<c0149e7e>] do_stop+0xe/0x70 (32)
> >  [<c0139c5a>] kthread+0xba/0xc0 (16)
> >  [<c0139ba0>] kthread+0x0/0xc0 (28)
> >  [<c0101385>] kernel_thread_helper+0x5/0x10 (16)
> 
> I replaced this with raw_local_irq_disable and it really locks up 
> everytime now.  I need to look into how this is best done.

how come it is stopping the machine during bootup? Or does the lockup 
occur during shutdown? changing the local_irq_disable() to 
raw_local_irq_disable() looks wrong because sooner or later we hit 
complete(). You are probably locking up earlier than that though, 
perhaps in stopmachine_set_state()?

but stop_machine() looks quite preempt-unsafe to begin with. The 
local_irq_disable() would not be needed at all if prior the 
for_each_online_cpu() loop we'd use set_cpus_allowed. The current method 
of achieving 'no preemption' is simply racy even during normal 
CONFIG_PREEMPT.

	Ingo

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 08:47 +0200, Ingo Molnar wrote:

> 
> unfortunately the space of "patches that break the kernel" is infinitely 
> larger (and infinitely easier to generate) than the space of "patches 
> that improve the kernel" - so i'll skip your patch for now ;-)

Yeah, I took out my "bad" fix and it still locks up. Since I got the NMI
working I was able to get output. Unfortunately, the laptop doesn't have
a serial (damn vendors getting rid of the very useful _for us_ uart).
And so far the netconsole doesn't seem to be working yet. 

Here's the output (typed from screen).

Freeing unused kernel memory: 296k freed
NMI watchdog detected lockup on CPU#1 (50000/50000)

Pid: 796, comm:  hotplug
EIP: 0060:[<c0331662>] CPU: 1
EIP is at _raw_spin_lock+0x72/0xa0
EFLAGS: 00000002 Not tainted (2.6.13-rc6-rt6)
EXI: 00000001 EBX: .... (I'm not about to type all the registers out).
[...]
try_to_wake_up+0x4e
__reacquire_kernel_lock+0x2d
pick_new_owner+0x6b
wake_up_process+0x30
rt_up+0x290
chrdev_open+0xfd
lock_kernel+0x28
chrdev_open+0xfd
tty_open+0x0
chrdev_open+0xfd
file_move+0x20
dentry_open+0x17a
filp_open+0x68
_spin_lock+0x23
get_unused_fd+0xa2
sys_open+0x4f
syscall_call+0x7
----
preempt count: 3
3-level deep critical section nesting:
----
rt_up+0x2c
add_preempt_count+0x1a
add_preempt_count+0x1a
----
showing all locks held by: (hotplug/796 [...]) 

- No locks where shown.


> 
> how come it is stopping the machine during bootup? Or does the lockup 
> occur during shutdown? changing the local_irq_disable() to 
> raw_local_irq_disable() looks wrong because sooner or later we hit 
> complete(). You are probably locking up earlier than that though, 
> perhaps in stopmachine_set_state()?

This is on bootup.  The stopmachine is used in sys_init_module. 

> 
> but stop_machine() looks quite preempt-unsafe to begin with. The 
> local_irq_disable() would not be needed at all if prior the 
> for_each_online_cpu() loop we'd use set_cpus_allowed. The current method 
> of achieving 'no preemption' is simply racy even during normal 
> CONFIG_PREEMPT.

I guess I'll need to look into this further.

-- Steve

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 10:05 -0400, Steven Rostedt wrote:
> On Wed, 2005-08-17 at 08:47 +0200, Ingo Molnar wrote:
> 
> > 
> > unfortunately the space of "patches that break the kernel" is infinitely 
> > larger (and infinitely easier to generate) than the space of "patches 
> > that improve the kernel" - so i'll skip your patch for now ;-)
> 
> Yeah, I took out my "bad" fix and it still locks up. Since I got the NMI
> working I was able to get output. Unfortunately, the laptop doesn't have
> a serial (damn vendors getting rid of the very useful _for us_ uart).
> And so far the netconsole doesn't seem to be working yet. 
> 

Silly me, I never compiled netconsole into the kernel for my laptop
(I've done it on all my other machines that happen to have a serial!).

OK the output from netconsole still seems like netconsole itself is
causing some problems.  But I think it is also showing this lockup. I'll
recompile my kernel as UP and see if netconsole works fine.

Here's what netconsole shows:

netconsole: device eth0 not up yet, forcing it
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is on for TX and on for RX.
softirq-net-rx//6[CPU#0]: BUG in up_mutex at kernel/rt.c:1945
 [<c012258c>] __WARN_ON+0x6c/0x90 (8)
 [<c02d3c4a>] net_rx_action+0x15a/0x1e0 (44)
 [<c01406e0>] up_mutex+0x410/0x440 (4)
 [<c02d3c4a>] net_rx_action+0x15a/0x1e0 (40)
 [<c0127b40>] ksoftirqd+0xf0/0x170 (40)
 [<c0127a50>] ksoftirqd+0x0/0x170 (32)
 [<c0139c5a>] kthread+0xba/0xc0 (4)
 [<c0139ba0>] kthread+0x0/0xc0 (28)
 [<c0101385>] kernel_thread_helper+0x5/0x10 (16)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c014201a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)

[... a bunch more dumps ...]

netconsole: network logging started
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
ICH4: IDE controller at PCI slot 0000:00:1f.1

[... normal output ...]

Freeing unused kernel memory: 296k freed
BUG: using smp_processor_id() in preemptible [00000000] code:
kjournald/795
caller is netpoll_send_skb+0x38/0x100
 [<c020bb74>] debug_smp_processor_id+0xb4/0xc0 (8)
 [<c02e1e48>] netpoll_send_skb+0x38/0x100 (8)
 [<c02e1e48>] netpoll_send_skb+0x38/0x100 (20)
 [<c02ad327>] write_msg+0x67/0xd0 (24)
 [<c0121962>] __call_console_drivers+0x62/0x70 (32)
 [<c0121ab3>] call_console_drivers+0xb3/0x150 (28)
 [<c0121f81>] release_console_sem+0x51/0xf0 (36)
 [<c0121dda>] vprintk+0x18a/0x240 (28)
 [<c011c05b>] __wake_up+0x4b/0x80 (76)
 [<c0121c47>] printk+0x17/0x20 (36)
 [<c01d4e01>] kjournald+0x91/0x250 (12)
 [<c01420ca>] sub_preempt_count+0x1a/0x20 (28)
 [<c0333502>] _raw_spin_unlock+0x12/0x30 (12)
 [<c01420ca>] sub_preempt_count+0x1a/0x20 (20)
 [<c011a507>] schedule_tail+0x87/0x110 (12)
 [<c01d4d50>] commit_timeout+0x0/0x10 (32)
 [<c01d4d70>] kjournald+0x0/0x250 (32)
 [<c0101385>] kernel_thread_helper+0x5/0x10 (16)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c014201a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)

[ looks like the above is the NMI going off and netconsole screaming ]

[... a bunch more smp_processor_id() bug outputs ...]
[ then ...]
Hotplug/796[CPU#1]: BUG in set_new_owner at kernel/rt.c:916
NMI watchdog detected lockup on CPU#1 (50000/50000)

Pid: 796, comm:              hotplug

----

Then no more output.  So netconsole sort of works, but seems to croak on
a NMI. It may be best if I keep netconsole off. Or do you have any
better ideas?

> Here's the output (typed from screen).
> 
> Freeing unused kernel memory: 296k freed
> NMI watchdog detected lockup on CPU#1 (50000/50000)
> 
> Pid: 796, comm:  hotplug
> EIP: 0060:[<c0331662>] CPU: 1
> EIP is at _raw_spin_lock+0x72/0xa0
> EFLAGS: 00000002 Not tainted (2.6.13-rc6-rt6)
> EXI: 00000001 EBX: .... (I'm not about to type all the registers out).
> [...]
> try_to_wake_up+0x4e
> __reacquire_kernel_lock+0x2d
> pick_new_owner+0x6b
> wake_up_process+0x30
> rt_up+0x290
> chrdev_open+0xfd
> lock_kernel+0x28
> chrdev_open+0xfd
> tty_open+0x0
> chrdev_open+0xfd
> file_move+0x20
> dentry_open+0x17a
> filp_open+0x68
> _spin_lock+0x23
> get_unused_fd+0xa2
> sys_open+0x4f
> syscall_call+0x7
> ----
> preempt count: 3
> 3-level deep critical section nesting:
> ----
> rt_up+0x2c
> add_preempt_count+0x1a
> add_preempt_count+0x1a
> ----
> showing all locks held by: (hotplug/796 [...]) 
> 
> - No locks where shown.
> 

I kept the above just to see what is shown without netconsole, but I
only get a screen shot with it.

If worse comes to worse, I'll switch the testing to my main machine (AMD
SMP) and see if it crashes. It's just that this is the machine that I do
my email with. But at least it has a serial.

-- Steve

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 10:24 -0400, Steven Rostedt wrote:

> OK the output from netconsole still seems like netconsole itself is
> causing some problems.  But I think it is also showing this lockup. I'll
> recompile my kernel as UP and see if netconsole works fine.

Well, the UP kernel boots on my laptop, but netconsole gives strange
warnings. 

OK, what's the scoop with the illegal_API_call?  What is it about, and
what is the expected work around?

I'm also getting the following output on shutdown:

NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP ver 2.7
Bluetooth: L2CAP socket layer initialized
Bluetooth: RFCOMM ver 1.5
Bluetooth: RFCOMM socket layer initialized
Bluetooth: RFCOMM TTY layer initialized
BUG: nonzero lock count 1 at exit time?
            nfsd: 4696 [f7183830, 115]
 [<c0136922>] check_no_held_locks+0x62/0x330 (8)
 [<c011df67>] do_exit+0x257/0x480 (32)
 [<c013d052>] __module_put_and_exit+0x52/0x70 (40)
 [<f8d54583>] nfsd+0x2b3/0x340 [nfsd] (12)
 [<f8d542d0>] nfsd+0x0/0x340 [nfsd] (48)
 [<c010140d>] kernel_thread_helper+0x5/0x18 (16)
---------------------------
| preempt count: 00000000 ]
| 0-level deep critical section nesting:
----------------------------------------

------------------------------
| showing all locks held by: |  (nfsd/4696 [f7183830, 116]):
------------------------------

#001:             [c038e184] {kernel_sem.lock}
... acquired at:               lock_kernel+0x21/0x40

BUG: nfsd/4696, BKL held at task exit time!


And it goes on and on. This happens everytime. Without netconsole, I
only get the nonzero lock count error. Also, one of my lockups on SMP
had to do with the kernel_thread_helper:

Using IPI Shortcut mode
khelper/794[CPU#0]: BUG in set_new_owner at kernel/rt.c:916
NMI watchdog detected lockup on CPU#1 (50000/50000)

Pid: 24, comm:              khelper
EIP: 0060:[<c0140368>] CPU: 1
EIP is at up_mutex+0x98/0x440
 EFLAGS: 00000082    Not tainted  (2.6.13-rc6-rt6)
EAX: c04278a4 EBX: cf68feec ECX: 00000206 EDX: c0396b8c
ESI: cf68fec8 EDI: 00000246 EBP: c011c191 DS: 007b ES: 007b
CR0: 8005003b CR2: 00000000 CR3: 00478000 CR4: 000006d0
 [<c011c191>] complete+0x51/0x80 (40)
 [<c0133fef>] worker_thread+0x1cf/0x290 (40)
 [<c0133a60>] __call_usermodehelper+0x0/0x70 (24)
 [<c011bf70>] default_wake_function+0x0/0x30 (28)
 [<c011bf70>] default_wake_function+0x0/0x30 (32)
 [<c032fc6b>] schedule+0x4b/0x120 (12)
 [<c0133e20>] worker_thread+0x0/0x290 (24)
 [<c0139c5a>] kthread+0xba/0xc0 (4)
 [<c0139ba0>] kthread+0x0/0xc0 (28)
 [<c0101385>] kernel_thread_helper+0x5/0x10 (16)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c014201a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)

------------------------------
| showing all locks held by: |  (khelper/24 [c19df120, 111]):
------------------------------

NMI watchdog detected lockup on CPU#0 (50000/50000)

Pid: 794, comm:              khelper
EIP: 0060:[<c013ef1f>] CPU: 0
EIP is at __down_trylock+0x9f/0x330
 EFLAGS: 00000082    Not tainted  (2.6.13-rc6-rt6)
EAX: c0395238 EBX: 00000000 ECX: 00008000 EDX: 000043b0
ESI: c0395224 EDI: cf4d2000 EBP: 00000086 DS: 007b ES: 007b
CR0: 8005003b CR2: fff3f000 CR3: 00478000 CR4: 000006d0
 [<c0141033>] rt_down_trylock+0x33/0x480 (44)
 [<c0121db2>] vprintk+0x162/0x240 (8)
 [<c020986b>] vscnprintf+0x2b/0x40 (8)
 [<c0121db2>] vprintk+0x162/0x240 (24)
 [<c014201a>] add_preempt_count+0x1a/0x20 (36)
 [<c0333059>] _raw_spin_lock+0x19/0xa0 (12)
 [<c01420ca>] sub_preempt_count+0x1a/0x20 (12)
 [<c0141f27>] add_preempt_count_ti+0x27/0x100 (4)
 [<c014201a>] add_preempt_count+0x1a/0x20 (28)
 [<c0121c47>] printk+0x17/0x20 (20)
 [<c0122587>] __WARN_ON+0x67/0x90 (12)
 [<c033100a>] __down_mutex+0x2fa/0x5d0 (48)
 [<c01247f0>] do_exit+0x170/0x490 (112)
 [<c01419cd>] atomic_dec_and_spin_lock+0x2d/0x80 (8)
 [<c01247f0>] do_exit+0x170/0x490 (8)
 [<c01247f0>] do_exit+0x170/0x490 (20)
 [<c01339a8>] ____call_usermodehelper+0xe8/0xf0 (40)
 [<c01338c0>] ____call_usermodehelper+0x0/0xf0 (12)
 [<c0101385>] kernel_thread_helper+0x5/0x10 (12)
---------------------------
| preempt count: 00000003 ]
| 3-level deep critical section nesting:
----------------------------------------
.. [<c014201a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)
.. [<c014201a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)
.. [<c014201a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)

------------------------------
| showing all locks held by: |  (khelper/794 [cf4f3740, 112]):
------------------------------


-- Steve

Re: 2.6.13-rc6-rt6

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Wed, 2005-08-17 at 10:24 -0400, Steven Rostedt wrote:
> 
> > OK the output from netconsole still seems like netconsole itself is
> > causing some problems.  But I think it is also showing this lockup. I'll
> > recompile my kernel as UP and see if netconsole works fine.
> 
> Well, the UP kernel boots on my laptop, but netconsole gives strange 
> warnings.
> 
> OK, what's the scoop with the illegal_API_call?  What is it about, and 
> what is the expected work around?

this is a recent change: i've started flagging "naked" use of 
local_irq_disable(), because it's a problem on PREEMPT_RT and it's a 
potential SMP bug on upstream kernels. A local_irq_disable() is 
converted either to raw_local_irq_disable() when justified (it's mostly 
only justified for lowlevel arch code), or is eliminated totally.  
(either by merging it into a nearby spin_lock API call, or by removing 
it altogether, making sure that code doesnt break).

Right now we print a warning on the first such API use, and then shut up 
about it. All local_irq_() APIs map to NOPs. (we keep the PF_IRQSOFF 
flag for compatibility, but only to get irqs_off() right which in turn 
shuts off a number of BUG_ON(!irqs_disabled()) warnings, and it doesnt 
have any other functional purpose.)

the desired end-result would be the total elimination of local_irq_*() 
API calls.

> I'm also getting the following output on shutdown:
> 
> NET: Registered protocol family 31
> Bluetooth: HCI device and connection manager initialized
> Bluetooth: HCI socket layer initialized
> Bluetooth: L2CAP ver 2.7
> Bluetooth: L2CAP socket layer initialized
> Bluetooth: RFCOMM ver 1.5
> Bluetooth: RFCOMM socket layer initialized
> Bluetooth: RFCOMM TTY layer initialized
> BUG: nonzero lock count 1 at exit time?
>             nfsd: 4696 [f7183830, 115]
>  [<c0136922>] check_no_held_locks+0x62/0x330 (8)
>  [<c011df67>] do_exit+0x257/0x480 (32)
>  [<c013d052>] __module_put_and_exit+0x52/0x70 (40)
>  [<f8d54583>] nfsd+0x2b3/0x340 [nfsd] (12)
>  [<f8d542d0>] nfsd+0x0/0x340 [nfsd] (48)
>  [<c010140d>] kernel_thread_helper+0x5/0x18 (16)
> ---------------------------
> | preempt count: 00000000 ]
> | 0-level deep critical section nesting:
> ----------------------------------------
> 
> ------------------------------
> | showing all locks held by: |  (nfsd/4696 [f7183830, 116]):
> ------------------------------
> 
> #001:             [c038e184] {kernel_sem.lock}
> ... acquired at:               lock_kernel+0x21/0x40
> 
> BUG: nfsd/4696, BKL held at task exit time!

hm, it seems nfsd forgets to do an unlock_kernel() in some exit path it 
seems? We are enforcing strict balanced lock use in PREEMPT_RT - the 
upstream kernel is more relaxed about it.

> And it goes on and on. This happens everytime. Without netconsole, I
> only get the nonzero lock count error. Also, one of my lockups on SMP
> had to do with the kernel_thread_helper:
> 
> Using IPI Shortcut mode
> khelper/794[CPU#0]: BUG in set_new_owner at kernel/rt.c:916

this is a 'must not happen'. Somehow lock->held list got non-empty.  
Maybe some use-after-free thing? Havent seen it myself.

	Ingo

Re: 2.6.13-rc6-rt6

[K.R. Foley]

Ingo Molnar wrote:
> * Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> 
>>On Wed, 2005-08-17 at 10:24 -0400, Steven Rostedt wrote:
>>
>>
>>>OK the output from netconsole still seems like netconsole itself is
>>>causing some problems.  But I think it is also showing this lockup. I'll
>>>recompile my kernel as UP and see if netconsole works fine.
>>
>>Well, the UP kernel boots on my laptop, but netconsole gives strange 
>>warnings.
>>
>>OK, what's the scoop with the illegal_API_call?  What is it about, and 
>>what is the expected work around?
> 
> 
> this is a recent change: i've started flagging "naked" use of 
> local_irq_disable(), because it's a problem on PREEMPT_RT and it's a 
> potential SMP bug on upstream kernels. A local_irq_disable() is 
> converted either to raw_local_irq_disable() when justified (it's mostly 
> only justified for lowlevel arch code), or is eliminated totally.  
> (either by merging it into a nearby spin_lock API call, or by removing 
> it altogether, making sure that code doesnt break).
> 
> Right now we print a warning on the first such API use, and then shut up 
> about it. All local_irq_() APIs map to NOPs. (we keep the PF_IRQSOFF 
> flag for compatibility, but only to get irqs_off() right which in turn 
> shuts off a number of BUG_ON(!irqs_disabled()) warnings, and it doesnt 
> have any other functional purpose.)
> 
> the desired end-result would be the total elimination of local_irq_*() 
> API calls.
> 
> 
>>I'm also getting the following output on shutdown:
>>
>>NET: Registered protocol family 31
>>Bluetooth: HCI device and connection manager initialized
>>Bluetooth: HCI socket layer initialized
>>Bluetooth: L2CAP ver 2.7
>>Bluetooth: L2CAP socket layer initialized
>>Bluetooth: RFCOMM ver 1.5
>>Bluetooth: RFCOMM socket layer initialized
>>Bluetooth: RFCOMM TTY layer initialized
>>BUG: nonzero lock count 1 at exit time?
>>            nfsd: 4696 [f7183830, 115]
>> [<c0136922>] check_no_held_locks+0x62/0x330 (8)
>> [<c011df67>] do_exit+0x257/0x480 (32)
>> [<c013d052>] __module_put_and_exit+0x52/0x70 (40)
>> [<f8d54583>] nfsd+0x2b3/0x340 [nfsd] (12)
>> [<f8d542d0>] nfsd+0x0/0x340 [nfsd] (48)
>> [<c010140d>] kernel_thread_helper+0x5/0x18 (16)
>>---------------------------
>>| preempt count: 00000000 ]
>>| 0-level deep critical section nesting:
>>----------------------------------------
>>
>>------------------------------
>>| showing all locks held by: |  (nfsd/4696 [f7183830, 116]):
>>------------------------------
>>
>>#001:             [c038e184] {kernel_sem.lock}
>>... acquired at:               lock_kernel+0x21/0x40
>>
>>BUG: nfsd/4696, BKL held at task exit time!
> 



> 
> hm, it seems nfsd forgets to do an unlock_kernel() in some exit path it 
> seems? We are enforcing strict balanced lock use in PREEMPT_RT - the 
> upstream kernel is more relaxed about it.
> 

This one has been biting me in the shorts since going to the 2.6.13-rc? 
RT series on my older SMP system at home. In every case the system hangs 
on shutdown and requires a hard reset. I just hadn't had the time to 
check into it. I was just in the process of building 2.6.13-rc6 without 
RT to check if it still happened. Guess I won't bother now. :-)

Aug 16 11:11:09 porky kernel: BUG: nonzero lock count 1 at exit time?
Aug 16 11:11:09 porky kernel:             nfsd: 4476 [dd1691a0, 115]
Aug 16 11:11:09 porky kernel:  [<c010418e>] dump_stack+0x1e/0x20 (20)
Aug 16 11:11:09 porky kernel:  [<c013b7ff>] 
check_no_held_locks+0x1af/0x370 (36)
Aug 16 11:11:09 porky kernel:  [<c0122e3f>] do_exit+0x26f/0x480 (44)
Aug 16 11:11:09 porky kernel:  [<c01413c1>] 
__module_put_and_exit+0x51/0x70 (16)
Aug 16 11:11:09 porky kernel:  [<e5a8558d>] nfsd+0x2bd/0x340 [nfsd] (68)
Aug 16 11:11:09 porky kernel:  [<c0101315>] 
kernel_thread_helper+0x5/0x10 (65485
2124)
Aug 16 11:11:09 porky kernel: ---------------------------
Aug 16 11:11:09 porky kernel: | preempt count: 00000000 ]
Aug 16 11:11:09 porky kernel: | 0-level deep critical section nesting:
Aug 16 11:11:09 porky kernel: ----------------------------------------
Aug 16 11:11:09 porky kernel:
Aug 16 11:11:09 porky kernel: ------------------------------
Aug 16 11:11:09 porky kernel: | showing all locks held by: |  (nfsd/4476 
[dd1691
a0, 116]):
Aug 16 11:11:09 porky kernel: ------------------------------
Aug 16 11:11:09 porky kernel:
Aug 16 11:11:09 porky kernel: #001:             [c0390fe4] {kernel_sem.lock}
Aug 16 11:11:09 porky kernel: ... acquired at: 
lock_kernel+0x28/0x50
Aug 16 11:11:09 porky kernel:
Aug 16 11:11:09 porky kernel: BUG: nfsd/4476, BKL held at task exit time!
Aug 16 11:11:09 porky kernel: BKL acquired at: nfsd+0x273/0x340 [nfsd]
Aug 16 11:11:09 porky kernel:  [c0390fe4] {kernel_sem.lock}
Aug 16 11:11:09 porky kernel: .. held by:              nfsd: 4476 
[dd1691a0, 116]
Aug 16 11:11:09 porky kernel: ... acquired at: 
lock_kernel+0x28/0x50
Aug 16 11:46:44 porky syslogd 1.4.1: restart.


> 
>>And it goes on and on. This happens everytime. Without netconsole, I
>>only get the nonzero lock count error. Also, one of my lockups on SMP
>>had to do with the kernel_thread_helper:
>>
>>Using IPI Shortcut mode
>>khelper/794[CPU#0]: BUG in set_new_owner at kernel/rt.c:916
> 
> 
> this is a 'must not happen'. Somehow lock->held list got non-empty.  
> Maybe some use-after-free thing? Havent seen it myself.
> 
> 	Ingo
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
    kr

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 08:47 +0200, Ingo Molnar wrote:

> but stop_machine() looks quite preempt-unsafe to begin with. The 
> local_irq_disable() would not be needed at all if prior the 
> for_each_online_cpu() loop we'd use set_cpus_allowed. The current method 
> of achieving 'no preemption' is simply racy even during normal 
> CONFIG_PREEMPT.

The code does look flakey, but I think it still works, and it may need
to have a raw_local_irq_disable.

The do_stop is bound to a CPU when it was created, so it doesn't need to
have a set_cpus_allowed.

I guess this is what is happening:

--- 

You start do_stop in a thread on the cpu that you want to run a function
on. (it's bound to that cpu)

do_stop creates a thread on all the other cpus that will run
stopmachine.

While it creates the threads, the ones that are already created yield in
case a thread wakes up on its cpu, so that that thread can migrate to
its own cpu. (all the threads are at FIFO MAX_RT_PRIO-1, so they should
never be preempted.

After all the threads are created and acknowledge themselves, the
do_stop changes the state to PREPARE.

In stopmachine, when PREPARE is seen, it turns off preemption and stops
yielding.

do_stop then changes the state to DISABLE_IRQ.

In stopmachine, when DISABLE_IRQ is seen, it disables IRQs.

Then the do_stop runs the function that is expected to run in the
STOPPED STATE.

---

Notes:

Each time the state is changed, the ack counter is zeroed and it wont
continue until all the threads acknowledge they hit the expected point.

I'm currious why it needs to go to preempt disable before going to
irqs_disabled? It seems that once it is at the point to go, all
processes should be locked on their CPUS and it would only need to goto
irqs_disable.

I'm also assuming that whatever function is being run expects to have
interrupts off on all CPUs. So a raw_local_irq_disable may be in order.

The comment above the stop_machine (called by do_stop) local_irq_disable
looks incorrect. It says 
/* Don't schedule us away at this point, please */  I don't see how it
can be scheduled out if it is running FIFO MAX_RT_PRIO-1.  It just
assume that it doesn't want any interrupts to go off.

Do you still see a problem with stop_machine?

-- Steve

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 12:10 -0500, K.R. Foley wrote:

> This one has been biting me in the shorts since going to the 2.6.13-rc? 
> RT series on my older SMP system at home. In every case the system hangs 
> on shutdown and requires a hard reset. I just hadn't had the time to 
> check into it. I was just in the process of building 2.6.13-rc6 without 
> RT to check if it still happened. Guess I won't bother now. :-)

On my AMD SMP box, the system boots with no problems. It did lock up on
shutdown right after it showed the nfsd bug.  My laptop still locks up
on start-up  so I need to take a look at it.  It's hyper-threaded and
not normal SMP, although I don't think that would make much of a
difference.  I'll copy over my config of my AMD box to my laptop, and
make just the changes that are required for the hardware, and see if it
boots farther.

When I ran my laptop in UP, on shutdown it also saw the nfsd problem,
but it never locked up.  My AMD SMP box did lock up.

-- Steve

Re: 2.6.13-rc6-rt6

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Wed, 2005-08-17 at 10:24 -0400, Steven Rostedt wrote:
> 
> > OK the output from netconsole still seems like netconsole itself is
> > causing some problems.  But I think it is also showing this lockup. I'll
> > recompile my kernel as UP and see if netconsole works fine.
> 
> Well, the UP kernel boots on my laptop, but netconsole gives strange 
> warnings.

Now that printk is in essence preemptible, there shouldnt be any 
warnings from netconsole - if there are any then it should be possible 
to fix them.

	Ingo

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 14:31 -0400, Steven Rostedt wrote:

> On my AMD SMP box, the system boots with no problems. 

I shouldn't say no problems. I got this really nasty stuff on the serial
but not on the vga.  Below is the patch.

I'm currently compiling my laptop to try this out. I started converting
to my AMD box and decided to try this first. Unfortunately, I modified
enough to do a full compile from scratch again.


BUG: stack overflow: only 984 bytes left! [c05c0424...(c05c0000-c05c2000)]
----------------------------->
| new stack-footprint maximum: swapper/0, 7096 bytes (out of 8136 bytes).
------------|
 [<c05c2000>] no_halt+0x0/0x20 (20)
 [<c01430cb>] __mcount+0x4b/0xe0 (20)
 [<c0142a77>] fill_worst_stack+0x67/0x70 (12)
 [<c0142c30>] debug_stackoverflow+0x70/0xd0 (16)
 [<c01223d0>] printk+0x20/0x30 (8)
 [<c01223ee>] vprintk+0xe/0x2b0 (4)
 [<c0113d8c>] mcount+0x14/0x18 (20)
 [<c01223ee>] vprintk+0xe/0x2b0 (20)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (8)
 [<c01223bb>] printk+0xb/0x30 (60)
 [<c0113d8c>] mcount+0x14/0x18 (20)
 [<c01223d0>] printk+0x20/0x30 (20)
 [<c0142aab>] __print_worst_stack+0x2b/0xf0 (20)
 [<c01457ac>] sub_preempt_count+0x1c/0x20 (8)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (24)
 [<c0142baf>] print_worst_stack+0x3f/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (4)
 [<c01425fb>] check_raw_flags+0xb/0x60 (4)
 [<c01430cb>] __mcount+0x4b/0xe0 (32)
 [<c02ec334>] _raw_spin_lock_irqsave+0x14/0xb0 (40)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (12)
 [<c0113d8c>] mcount+0x14/0x18 (8)
 [<c02ec334>] _raw_spin_lock_irqsave+0x14/0xb0 (20)
 [<c01425fb>] check_raw_flags+0xb/0x60 (4)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (12)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c01225dd>] vprintk+0x1fd/0x2b0 (12)
 [<c01430cb>] __mcount+0x4b/0xe0 (28)
 [<c01223bb>] printk+0xb/0x30 (40)
 [<c0113d8c>] mcount+0x14/0x18 (20)
 [<c01223bb>] printk+0xb/0x30 (20)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (20)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (32)
 [<c0142baf>] print_worst_stack+0x3f/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (4)
 [<c01425fb>] check_raw_flags+0xb/0x60 (4)
 [<c01430cb>] __mcount+0x4b/0xe0 (32)
 [<c02ec334>] _raw_spin_lock_irqsave+0x14/0xb0 (40)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (12)
 [<c0113d8c>] mcount+0x14/0x18 (8)
 [<c02ec334>] _raw_spin_lock_irqsave+0x14/0xb0 (20)
 [<c01425fb>] check_raw_flags+0xb/0x60 (4)
[<c01425fb>] check_raw_flags+0xb/0x60 (4)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (12)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c01225dd>] vprintk+0x1fd/0x2b0 (12)
 [<c01430cb>] __mcount+0x4b/0xe0 (28)
 [<c01223bb>] printk+0xb/0x30 (40)
 [<c0113d8c>] mcount+0x14/0x18 (20)
 [<c01223bb>] printk+0xb/0x30 (20)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (20)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (32)
 [<c0142baf>] print_worst_stack+0x3f/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (4)
 [<c01425fb>] check_raw_flags+0xb/0x60 (4)
 [<c01430cb>] __mcount+0x4b/0xe0 (32)
 [<c02ec334>] _raw_spin_lock_irqsave+0x14/0xb0 (40)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (12)
 [<c0113d8c>] mcount+0x14/0x18 (8)
 [<c02ec334>] _raw_spin_lock_irqsave+0x14/0xb0 (20)
 [<c01425fb>] check_raw_flags+0xb/0x60 (4)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (12)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c01225dd>] vprintk+0x1fd/0x2b0 (12)
 [<c01430cb>] __mcount+0x4b/0xe0 (28)
 [<c01223bb>] printk+0xb/0x30 (40)
 [<c0113d8c>] mcount+0x14/0x18 (20)
 [<c01223bb>] printk+0xb/0x30 (20)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (20)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (32)
 [<c0142baf>] print_worst_stack+0x3f/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (4)
 [<c01425fb>] check_raw_flags+0xb/0x60 (4)
 [<c01430cb>] __mcount+0x4b/0xe0 (32)
 [<c02ec334>] _raw_spin_lock_irqsave+0x14/0xb0 (40)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (12)
 [<c0113d8c>] mcount+0x14/0x18 (8)
 [<c02ec334>] _raw_spin_lock_irqsave+0x14/0xb0 (20)
 [<c01425fb>] check_raw_flags+0xb/0x60 (4)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (12)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c01225dd>] vprintk+0x1fd/0x2b0 (12)
 [<c01430cb>] __mcount+0x4b/0xe0 (28)
 [<c01223bb>] printk+0xb/0x30 (40)
 [<c0113d8c>] mcount+0x14/0x18 (20)
 [<c01223bb>] printk+0xb/0x30 (20)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (20)
 [<c0142b8a>] print_worst_stack+0x1a/0x50 (32)
 [<c0142baf>] print_worst_stack+0x3f/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (4)
 [<c01425fb>] check_raw_flags+0xb/0x60 (4)
 [<c01430cb>] __mcount+0x4b/0xe0 (32)
 [<c02ec334>] _raw_spin_lock_irqsave+0x14/0xb0 (40)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (12)

[...]

 [<c01223bb>] printk+0xb/0x30 (40)
 [<c0113d8c>] mcount+0x14/0x18 (20)
 [<c01223bb>] printk+0xb/0x30 (20)
 [<c0142ae1>] __print_worst_stack+0x61/0xf0 (20)
 [<c0122753>] release_console_sem+0x33/0xf0 (32)
 [<c0142baf>] print_worst_stack+0x3f/0x50 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (8)
 [<c0142c17>] debug_stackoverflow+0x57/0xd0 (20)
 [<c01430cb>] __mcount+0x4b/0xe0 (20)
 [<c0122184>] call_console_drivers+0x14/0x150 (40)
 [<c0113d8c>] mcount+0x14/0x18 (20)
 [<c0122184>] call_console_drivers+0x14/0x150 (20)
 [<c0122753>] release_console_sem+0x33/0xf0 (44)
 [<c0122628>] vprintk+0x248/0x2b0 (32)
 [<c01223bb>] printk+0xb/0x30 (68)
 [<c0113d8c>] mcount+0x14/0x18 (20)
 [<c01223d0>] printk+0x20/0x30 (20)
 [<c05d2d4c>] sched_init+0x12c/0x150 (20)
 [<c0113d8c>] mcount+0x14/0x18 (8)
 [<c05c2901>] start_kernel+0x51/0x220 (32)



-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_goliath/kernel/latency.c
===================================================================
--- linux_realtime_goliath/kernel/latency.c	(revision 295)
+++ linux_realtime_goliath/kernel/latency.c	(working copy)
@@ -377,8 +377,11 @@
 		print_worst_stack();
 		tr->stack_check++;
 	} else
-		if (worst_stack_printed != worst_stack_left)
+		if (worst_stack_printed != worst_stack_left) {
+			tr->stack_check--;
 			print_worst_stack();
+			tr->stack_check++;
+		}
 out:
 	atomic_dec(&tr->disabled);
 }

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 13:32 -0400, Steven Rostedt wrote:
> On Wed, 2005-08-17 at 08:47 +0200, Ingo Molnar wrote:
> 
> > but stop_machine() looks quite preempt-unsafe to begin with. The 
> > local_irq_disable() would not be needed at all if prior the 
> > for_each_online_cpu() loop we'd use set_cpus_allowed. The current method 
> > of achieving 'no preemption' is simply racy even during normal 
> > CONFIG_PREEMPT.
> 
> The code does look flakey, but I think it still works, and it may need
> to have a raw_local_irq_disable.

I added this patch to my AMD box is it runs fine.  So I'm assuming that
we do actually want interrupts disabled here.

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_goliath/kernel/stop_machine.c
===================================================================
--- linux_realtime_goliath/kernel/stop_machine.c	(revision 295)
+++ linux_realtime_goliath/kernel/stop_machine.c	(working copy)
@@ -40,7 +40,7 @@
 	while (stopmachine_state != STOPMACHINE_EXIT) {
 		if (stopmachine_state == STOPMACHINE_DISABLE_IRQ 
 		    && !irqs_disabled) {
-			local_irq_disable();
+			raw_local_irq_disable();
 			irqs_disabled = 1;
 			/* Ack: irqs disabled. */
 			smp_mb(); /* Must read state first. */
@@ -66,7 +66,7 @@
 	atomic_inc(&stopmachine_thread_ack);
 
 	if (irqs_disabled)
-		local_irq_enable();
+		raw_local_irq_enable();
 	if (prepared)
 		preempt_enable();
 
@@ -120,7 +120,7 @@
 	}
 
 	/* Don't schedule us away at this point, please. */
-	local_irq_disable();
+	raw_local_irq_disable();
 
 	/* Now they are all started, make them hold the CPUs, ready. */
 	stopmachine_set_state(STOPMACHINE_PREPARE);
@@ -134,7 +134,7 @@
 static void restart_machine(void)
 {
 	stopmachine_set_state(STOPMACHINE_EXIT);
-	local_irq_enable();
+	raw_local_irq_enable();
 }
 
 struct stop_machine_data

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 21:27 +0200, Ingo Molnar wrote:

> 
> Now that printk is in essence preemptible, there shouldnt be any 
> warnings from netconsole - if there are any then it should be possible 
> to fix them.
> 

Then I guess write_msg in netconsole.c needs to remove all the
local_irq_disable and enables.

-- Steve

Re: 2.6.13-rc6-rt8

[Peter Bortas]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

2.6.13-rc6-rt8 fails to build with my configuration (attached):

net/built-in.o: In function `ip_rt_init':
: undefined reference to `__you_cannot_kmalloc_that_much'
make[1]: *** [.tmp_vmlinux1] Error 1
make[1]: Leaving directory `/usr/src/linux-2.6.13-rc6'
make: *** [stamp-build] Error 2

-- 
Peter Bortas                  http://peter.bortas.org


[-- Attachment #2: rc6-rt8 config --]
[-- Type: text/plain, Size: 56678 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.13-rc6-rt8.050817
# Wed Aug 17 21:16:42 2005
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
# CONFIG_CLEAN_COMPILE is not set
CONFIG_BROKEN=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_SYSCTL=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_CPUSETS is not set
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
CONFIG_X86_GENERIC=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
# CONFIG_LEGACY_TIMER is not set
CONFIG_HPET_TIMER=y
# CONFIG_HIGH_RES_TIMERS is not set
CONFIG_SMP=y
CONFIG_NR_CPUS=32
CONFIG_SCHED_SMT=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_STATS=y
CONFIG_RCU_TORTURE_TEST=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_ASM_SEMAPHORES=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_IOAPIC_FAST=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_NONFATAL is not set
# CONFIG_X86_MCE_P4THERMAL is not set
CONFIG_TOSHIBA=m
CONFIG_I8K=m
# CONFIG_X86_REBOOTFIXUPS is not set
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m

#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_HIGHMEM=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_HIGHPTE is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
CONFIG_IRQBALANCE=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_PHYSICAL_START=0x100000
# CONFIG_KEXEC is not set

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_SOFTWARE_SUSPEND=y
CONFIG_PM_STD_PARTITION=""

#
# ACPI (Advanced Configuration and Power Interface) Support
#
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
# CONFIG_ACPI_HOTKEY is not set
CONFIG_ACPI_FAN=m
CONFIG_ACPI_PROCESSOR=m
CONFIG_ACPI_THERMAL=m
CONFIG_ACPI_ASUS=m
# CONFIG_ACPI_IBM is not set
CONFIG_ACPI_TOSHIBA=m
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set

#
# APM (Advanced Power Management) BIOS Support
#
CONFIG_APM=m
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
CONFIG_APM_DO_ENABLE=y
# CONFIG_APM_CPU_IDLE is not set
# CONFIG_APM_DISPLAY_BLANK is not set
# CONFIG_APM_RTC_IS_GMT is not set
CONFIG_APM_ALLOW_INTS=y
# CONFIG_APM_REAL_MODE_POWER_OFF is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=m
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=m
# CONFIG_CPU_FREQ_STAT_DETAILS is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
# CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set

#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K6=m
CONFIG_X86_POWERNOW_K7=m
CONFIG_X86_POWERNOW_K7_ACPI=y
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_POWERNOW_K8_ACPI=y
CONFIG_X86_GX_SUSPMOD=m
CONFIG_X86_SPEEDSTEP_CENTRINO=m
CONFIG_X86_SPEEDSTEP_CENTRINO_ACPI=y
CONFIG_X86_SPEEDSTEP_CENTRINO_TABLE=y
CONFIG_X86_SPEEDSTEP_ICH=m
CONFIG_X86_SPEEDSTEP_SMI=m
CONFIG_X86_P4_CLOCKMOD=m
# CONFIG_X86_CPUFREQ_NFORCE2 is not set
CONFIG_X86_LONGRUN=m
CONFIG_X86_LONGHAUL=m

#
# shared options
#
CONFIG_X86_ACPI_CPUFREQ_PROC_INTF=y
CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_X86_SPEEDSTEP_RELAXED_CAP_CHECK=y

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
# CONFIG_PCI_DEBUG is not set
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
CONFIG_EISA=y
CONFIG_EISA_VLB_PRIMING=y
CONFIG_EISA_PCI_EISA=y
CONFIG_EISA_VIRTUAL_ROOT=y
CONFIG_EISA_NAMES=y
CONFIG_MCA=y
CONFIG_MCA_LEGACY=y
CONFIG_MCA_PROC_FS=y
CONFIG_SCx200=m
# CONFIG_HOTPLUG_CPU is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PCI Hotplug Support
#
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_COMPAQ=m
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not set
CONFIG_HOTPLUG_PCI_IBM=m
CONFIG_HOTPLUG_PCI_ACPI=m
# CONFIG_HOTPLUG_PCI_ACPI_IBM is not set
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_SHPC=m
# CONFIG_HOTPLUG_PCI_SHPC_POLL_EVENT_MODE is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_AOUT is not set
CONFIG_BINFMT_MISC=m

#
# Networking
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=m
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
CONFIG_NET_KEY=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
# CONFIG_IP_ROUTE_MULTIPATH is not set
# CONFIG_IP_ROUTE_VERBOSE is not set
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
# CONFIG_NET_IPGRE_BROADCAST is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_TUNNEL=m
CONFIG_IP_TCPDIAG=m
CONFIG_IP_TCPDIAG_IPV6=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_BIC=y

#
# IP: Virtual Server Configuration
#
CONFIG_IP_VS=m
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IPV6=m
CONFIG_IPV6_PRIVACY=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
CONFIG_INET6_TUNNEL=m
CONFIG_IPV6_TUNNEL=m
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_BRIDGE_NETFILTER=y

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
CONFIG_IP_NF_CT_ACCT=y
CONFIG_IP_NF_CONNTRACK_MARK=y
# CONFIG_IP_NF_CT_PROTO_SCTP is not set
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_PHYSDEV=m
# CONFIG_IP_NF_MATCH_ADDRTYPE is not set
# CONFIG_IP_NF_MATCH_REALM is not set
CONFIG_IP_NF_MATCH_SCTP=m
CONFIG_IP_NF_MATCH_COMMENT=m
CONFIG_IP_NF_MATCH_CONNMARK=m
CONFIG_IP_NF_MATCH_HASHLIMIT=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
CONFIG_IP_NF_TARGET_CONNMARK=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_TARGET_NOTRACK=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# IPv6: Netfilter Configuration (EXPERIMENTAL)
#
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_LIMIT=m
CONFIG_IP6_NF_MATCH_MAC=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_MULTIPORT=m
CONFIG_IP6_NF_MATCH_OWNER=m
CONFIG_IP6_NF_MATCH_MARK=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_AHESP=m
CONFIG_IP6_NF_MATCH_LENGTH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_PHYSDEV=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_TARGET_MARK=m
CONFIG_IP6_NF_RAW=m

#
# DECnet: Netfilter Configuration
#
CONFIG_DECNET_NF_GRABULATOR=m

#
# Bridge: Netfilter Configuration
#
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m

#
# SCTP Configuration (EXPERIMENTAL)
#
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
CONFIG_SCTP_HMAC_NONE=y
# CONFIG_SCTP_HMAC_SHA1 is not set
# CONFIG_SCTP_HMAC_MD5 is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
CONFIG_ATM_CLIP_NO_ICMP=y
CONFIG_ATM_LANE=m
CONFIG_ATM_MPOA=m
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_BRIDGE=m
CONFIG_VLAN_8021Q=m
CONFIG_DECNET=m
CONFIG_DECNET_ROUTER=y
CONFIG_DECNET_ROUTE_FWMARK=y
CONFIG_LLC=y
CONFIG_LLC2=m
CONFIG_IPX=m
CONFIG_IPX_INTERN=y
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=y
CONFIG_LTPC=m
CONFIG_COPS=m
CONFIG_COPS_DAYNA=y
CONFIG_COPS_TANGENT=y
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
CONFIG_X25=m
CONFIG_LAPB=m
# CONFIG_NET_DIVERT is not set
CONFIG_ECONET=m
CONFIG_ECONET_AUNUDP=y
CONFIG_ECONET_NATIVE=y
CONFIG_WAN_ROUTER=m
# CONFIG_NET_SCHED is not set
# CONFIG_NET_CLS_ROUTE is not set

#
# Network testing
#
CONFIG_NET_PKTGEN=m
CONFIG_HAMRADIO=y

#
# Packet Radio protocols
#
CONFIG_AX25=m
CONFIG_AX25_DAMA_SLAVE=y
CONFIG_NETROM=m
CONFIG_ROSE=m

#
# AX.25 network device drivers
#
CONFIG_MKISS=m
CONFIG_6PACK=m
CONFIG_BPQETHER=m
CONFIG_DMASCC=m
CONFIG_SCC=m
CONFIG_SCC_DELAY=y
CONFIG_SCC_TRXECHO=y
CONFIG_BAYCOM_SER_FDX=m
CONFIG_BAYCOM_SER_HDX=m
CONFIG_BAYCOM_PAR=m
CONFIG_BAYCOM_EPP=m
CONFIG_YAM=m
CONFIG_IRDA=m

#
# IrDA protocols
#
CONFIG_IRLAN=m
CONFIG_IRNET=m
CONFIG_IRCOMM=m
CONFIG_IRDA_ULTRA=y

#
# IrDA options
#
CONFIG_IRDA_CACHE_LAST_LSAP=y
CONFIG_IRDA_FAST_RR=y
# CONFIG_IRDA_DEBUG is not set

#
# Infrared-port device drivers
#

#
# SIR device drivers
#
CONFIG_IRTTY_SIR=m

#
# Dongle support
#
CONFIG_DONGLE=y
CONFIG_ESI_DONGLE=m
CONFIG_ACTISYS_DONGLE=m
CONFIG_TEKRAM_DONGLE=m
CONFIG_LITELINK_DONGLE=m
CONFIG_MA600_DONGLE=m
CONFIG_GIRBIL_DONGLE=m
CONFIG_MCP2120_DONGLE=m
CONFIG_OLD_BELKIN_DONGLE=m
CONFIG_ACT200L_DONGLE=m

#
# Old SIR device drivers
#
CONFIG_IRPORT_SIR=m

#
# Old Serial dongle support
#
CONFIG_DONGLE_OLD=y
CONFIG_ESI_DONGLE_OLD=m
CONFIG_ACTISYS_DONGLE_OLD=m
CONFIG_TEKRAM_DONGLE_OLD=m
CONFIG_GIRBIL_DONGLE_OLD=m
CONFIG_LITELINK_DONGLE_OLD=m
CONFIG_MCP2120_DONGLE_OLD=m
CONFIG_OLD_BELKIN_DONGLE_OLD=m
CONFIG_ACT200L_DONGLE_OLD=m
CONFIG_MA600_DONGLE_OLD=m

#
# FIR device drivers
#
CONFIG_USB_IRDA=m
CONFIG_SIGMATEL_FIR=m
CONFIG_NSC_FIR=m
CONFIG_WINBOND_FIR=m
CONFIG_TOSHIBA_FIR=m
CONFIG_SMC_IRCC_FIR=m
CONFIG_ALI_FIR=m
CONFIG_VLSI_FIR=m
CONFIG_VIA_FIR=m
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
# CONFIG_BT_HIDP is not set

#
# Bluetooth device drivers
#
CONFIG_BT_HCIUSB=m
CONFIG_BT_HCIUSB_SCO=y
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_BCSP_TXCRC=y
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIVHCI=m

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
# CONFIG_DEBUG_DRIVER is not set

#
# Memory Technology Devices (MTD)
#
CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not set
CONFIG_MTD_CONCAT=m
# CONFIG_MTD_PARTITIONS is not set

#
# User Modules And Translation Layers
#
CONFIG_MTD_CHAR=m
CONFIG_MTD_BLOCK=m
CONFIG_MTD_BLOCK_RO=m
CONFIG_FTL=m
CONFIG_NFTL=m
CONFIG_NFTL_RW=y
CONFIG_INFTL=m

#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=m
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
CONFIG_MTD_CFI_INTELEXT=m
CONFIG_MTD_CFI_AMDSTD=m
CONFIG_MTD_CFI_AMDSTD_RETRY=0
CONFIG_MTD_CFI_STAA=m
CONFIG_MTD_CFI_UTIL=m
CONFIG_MTD_RAM=m
CONFIG_MTD_ROM=m
CONFIG_MTD_ABSENT=m
# CONFIG_MTD_OBSOLETE_CHIPS is not set

#
# Mapping drivers for chip access
#
CONFIG_MTD_COMPLEX_MAPPINGS=y
CONFIG_MTD_PHYSMAP=m
CONFIG_MTD_PHYSMAP_START=0x8000000
CONFIG_MTD_PHYSMAP_LEN=0x4000000
CONFIG_MTD_PHYSMAP_BANKWIDTH=2
CONFIG_MTD_SC520CDP=m
CONFIG_MTD_AMD76XROM=m
# CONFIG_MTD_ICHXROM is not set
CONFIG_MTD_SCB2_FLASH=m
CONFIG_MTD_L440GX=m
CONFIG_MTD_PCI=m
# CONFIG_MTD_PLATRAM is not set

#
# Self-contained MTD device drivers
#
CONFIG_MTD_PMC551=m
CONFIG_MTD_PMC551_BUGFIX=y
# CONFIG_MTD_PMC551_DEBUG is not set
CONFIG_MTD_SLRAM=m
# CONFIG_MTD_PHRAM is not set
CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLKMTD=m
# CONFIG_MTD_BLOCK2MTD is not set

#
# Disk-On-Chip Device Drivers
#
CONFIG_MTD_DOC2000=m
CONFIG_MTD_DOC2001=m
CONFIG_MTD_DOC2001PLUS=m
CONFIG_MTD_DOCPROBE=m
CONFIG_MTD_DOCECC=m
# CONFIG_MTD_DOCPROBE_ADVANCED is not set
CONFIG_MTD_DOCPROBE_ADDRESS=0

#
# NAND Flash Device Drivers
#
CONFIG_MTD_NAND=m
# CONFIG_MTD_NAND_VERIFY_WRITE is not set
CONFIG_MTD_NAND_IDS=m
# CONFIG_MTD_NAND_DISKONCHIP is not set

#
# Parallel port support
#
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
CONFIG_PARPORT_PC_FIFO=y
CONFIG_PARPORT_PC_SUPERIO=y
CONFIG_PARPORT_NOT_PC=y
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_1284=y

#
# Plug and Play support
#
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set

#
# Protocols
#
CONFIG_ISAPNP=y
CONFIG_PNPBIOS=y
CONFIG_PNPBIOS_PROC_FS=y
CONFIG_PNPACPI=y

#
# Block devices
#
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_PS2=m
CONFIG_BLK_DEV_XD=m
CONFIG_PARIDE=m
CONFIG_PARIDE_PARPORT=m

#
# Parallel IDE high-level drivers
#
CONFIG_PARIDE_PD=m
CONFIG_PARIDE_PCD=m
CONFIG_PARIDE_PF=m
CONFIG_PARIDE_PT=m
CONFIG_PARIDE_PG=m

#
# Parallel IDE protocol modules
#
CONFIG_PARIDE_ATEN=m
CONFIG_PARIDE_BPCK=m
CONFIG_PARIDE_BPCK6=m
CONFIG_PARIDE_COMM=m
CONFIG_PARIDE_DSTR=m
CONFIG_PARIDE_FIT2=m
CONFIG_PARIDE_FIT3=m
CONFIG_PARIDE_EPAT=m
CONFIG_PARIDE_EPATC8=y
CONFIG_PARIDE_EPIA=m
CONFIG_PARIDE_FRIQ=m
CONFIG_PARIDE_FRPW=m
CONFIG_PARIDE_KBIC=m
CONFIG_PARIDE_KTTI=m
CONFIG_PARIDE_ON20=m
CONFIG_PARIDE_ON26=m
CONFIG_BLK_CPQ_DA=m
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_UMEM=m
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_LBD=y
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_ATA_OVER_ETH is not set

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDETAPE=m
CONFIG_BLK_DEV_IDEFLOPPY=m
CONFIG_BLK_DEV_IDESCSI=m
CONFIG_IDE_TASK_IOCTL=y

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_CMD640=y
CONFIG_BLK_DEV_CMD640_ENHANCED=y
CONFIG_BLK_DEV_IDEPNP=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
CONFIG_BLK_DEV_OPTI621=y
CONFIG_BLK_DEV_RZ1000=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
# CONFIG_IDEDMA_PCI_AUTO is not set
CONFIG_BLK_DEV_AEC62XX=y
CONFIG_BLK_DEV_ALI15X3=y
# CONFIG_WDC_ALI15X3 is not set
CONFIG_BLK_DEV_AMD74XX=y
CONFIG_BLK_DEV_ATIIXP=y
CONFIG_BLK_DEV_CMD64X=y
CONFIG_BLK_DEV_TRIFLEX=y
CONFIG_BLK_DEV_CY82C693=y
CONFIG_BLK_DEV_CS5520=y
CONFIG_BLK_DEV_CS5530=y
CONFIG_BLK_DEV_HPT34X=y
# CONFIG_HPT34X_AUTODMA is not set
CONFIG_BLK_DEV_HPT366=y
CONFIG_BLK_DEV_SC1200=y
CONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_IT821X is not set
CONFIG_BLK_DEV_NS87415=y
CONFIG_BLK_DEV_PDC202XX_OLD=y
# CONFIG_PDC202XX_BURST is not set
CONFIG_BLK_DEV_PDC202XX_NEW=y
CONFIG_PDC202XX_FORCE=y
CONFIG_BLK_DEV_SVWKS=y
CONFIG_BLK_DEV_SIIMAGE=y
CONFIG_BLK_DEV_SIS5513=y
CONFIG_BLK_DEV_SLC90E66=y
CONFIG_BLK_DEV_TRM290=y
CONFIG_BLK_DEV_VIA82CXXX=y
# CONFIG_IDE_ARM is not set
# CONFIG_IDE_CHIPSETS is not set
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_IVB=y
# CONFIG_IDEDMA_AUTO is not set
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_SCSI=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set

#
# SCSI Transport Attributes
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
# CONFIG_SCSI_ISCSI_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
CONFIG_SCSI_AHA152X=m
CONFIG_SCSI_AHA1542=m
CONFIG_SCSI_AHA1740=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=253
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
CONFIG_AIC7XXX_PROBE_EISA_VL=y
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=15000
CONFIG_AIC79XX_ENABLE_RD_STRM=y
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_SCSI_SATA is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_CPQFCTS is not set
CONFIG_SCSI_DMX3191D=m
CONFIG_SCSI_DTC3280=m
CONFIG_SCSI_EATA=m
# CONFIG_SCSI_EATA_TAGGED_QUEUE is not set
# CONFIG_SCSI_EATA_LINKED_COMMANDS is not set
CONFIG_SCSI_EATA_MAX_TAGS=16
CONFIG_SCSI_EATA_PIO=m
CONFIG_SCSI_FUTURE_DOMAIN=m
CONFIG_SCSI_FD_MCS=m
CONFIG_SCSI_GDTH=m
CONFIG_SCSI_GENERIC_NCR5380=m
CONFIG_SCSI_GENERIC_NCR5380_MMIO=m
CONFIG_SCSI_GENERIC_NCR53C400=y
CONFIG_SCSI_IBMMCA=m
CONFIG_IBMMCA_SCSI_ORDER_STANDARD=y
# CONFIG_IBMMCA_SCSI_DEV_RESET is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
CONFIG_SCSI_PPA=m
CONFIG_SCSI_IMM=m
CONFIG_SCSI_IZIP_EPP16=y
# CONFIG_SCSI_IZIP_SLOW_CTR is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_NCR_D700 is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_NCR_Q720 is not set
# CONFIG_SCSI_MCA_53C9X is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA2XXX=y
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA24XX is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_SEAGATE is not set
# CONFIG_SCSI_SIM710 is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
CONFIG_CD_NO_IDESCSI=y
CONFIG_AZTCD=m
CONFIG_GSCD=m
CONFIG_SBPCD=m
CONFIG_MCDX=m
CONFIG_OPTCD=m
CONFIG_CM206=m
CONFIG_SJCD=m
CONFIG_ISP16_CDI=m
CONFIG_CDU31A=m
CONFIG_CDU535=m

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
# CONFIG_MD_RAID10 is not set
CONFIG_MD_RAID5=m
CONFIG_MD_RAID6=m
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
# CONFIG_DM_SNAPSHOT is not set
# CONFIG_DM_MIRROR is not set
# CONFIG_DM_ZERO is not set
# CONFIG_DM_MULTIPATH is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set

#
# IEEE 1394 (FireWire) support
#
CONFIG_IEEE1394=m

#
# Subsystem Options
#
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
CONFIG_IEEE1394_OUI_DB=y
CONFIG_IEEE1394_EXTRA_CONFIG_ROMS=y
CONFIG_IEEE1394_CONFIG_ROM_IP1394=y
# CONFIG_IEEE1394_EXPORT_FULL_API is not set

#
# Device Drivers
#
CONFIG_IEEE1394_PCILYNX=m
CONFIG_IEEE1394_OHCI1394=m

#
# Protocol Drivers
#
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_SBP2=m
# CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
CONFIG_IEEE1394_ETH1394=m
CONFIG_IEEE1394_DV1394=m
CONFIG_IEEE1394_RAWIO=m
CONFIG_IEEE1394_CMP=m
CONFIG_IEEE1394_AMDTP=m

#
# I2O device support
#
CONFIG_I2O=m
CONFIG_I2O_EXT_ADAPTEC=y
# CONFIG_I2O_CONFIG is not set
CONFIG_I2O_BUS=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m

#
# Network device support
#
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_EQUALIZER=m
CONFIG_TUN=m
CONFIG_NET_SB1000=m

#
# ARCnet devices
#
CONFIG_ARCNET=m
CONFIG_ARCNET_1201=m
CONFIG_ARCNET_1051=m
CONFIG_ARCNET_RAW=m
# CONFIG_ARCNET_CAP is not set
CONFIG_ARCNET_COM90xx=m
CONFIG_ARCNET_COM90xxIO=m
CONFIG_ARCNET_RIM_I=m
CONFIG_ARCNET_COM20020=m
CONFIG_ARCNET_COM20020_ISA=m
CONFIG_ARCNET_COM20020_PCI=m

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_EL1=m
CONFIG_EL2=m
CONFIG_ELPLUS=m
CONFIG_EL16=m
CONFIG_EL3=m
CONFIG_3C515=m
CONFIG_ELMC=m
CONFIG_ELMC_II=m
CONFIG_VORTEX=m
CONFIG_TYPHOON=m
CONFIG_LANCE=m
CONFIG_NET_VENDOR_SMC=y
CONFIG_WD80x3=m
CONFIG_ULTRAMCA=m
CONFIG_ULTRA=m
CONFIG_ULTRA32=m
CONFIG_SMC9194=m
CONFIG_NET_VENDOR_RACAL=y
CONFIG_NI5010=m
CONFIG_NI52=m
CONFIG_NI65=m

#
# Tulip family network device support
#
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
# CONFIG_TULIP_MMIO is not set
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_AT1700=m
CONFIG_DEPCA=m
CONFIG_HP100=m
CONFIG_NET_ISA=y
CONFIG_E2100=m
CONFIG_EWRK3=m
CONFIG_EEXPRESS=m
CONFIG_EEXPRESS_PRO=m
CONFIG_HPLAN_PLUS=m
CONFIG_HPLAN=m
CONFIG_LP486E=m
CONFIG_ETH16I=m
CONFIG_NE2000=m
CONFIG_ZNET=m
CONFIG_SEEQ8005=m
CONFIG_SKMC=m
CONFIG_NE2_MCA=m
CONFIG_IBMLANA=m
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
CONFIG_AMD8111E_NAPI=y
CONFIG_ADAPTEC_STARFIRE=m
# CONFIG_ADAPTEC_STARFIRE_NAPI is not set
CONFIG_AC3200=m
CONFIG_APRICOT=m
CONFIG_B44=m
CONFIG_FORCEDETH=m
CONFIG_CS89x0=m
CONFIG_DGRS=m
CONFIG_EEPRO100=m
CONFIG_E100=m
CONFIG_LNE390=m
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_NE3210=m
CONFIG_ES3210=m
# CONFIG_8139CP is not set
CONFIG_8139TOO=m
# CONFIG_8139TOO_PIO is not set
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_SIS900=m
CONFIG_EPIC100=m
CONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not set
CONFIG_TLAN=m
CONFIG_VIA_RHINE=m
CONFIG_VIA_RHINE_MMIO=y
CONFIG_NET_POCKET=y
CONFIG_ATP=m
CONFIG_DE600=m
CONFIG_DE620=m

#
# Ethernet (1000 Mbit)
#
CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not set
CONFIG_DL2K=m
CONFIG_E1000=y
# CONFIG_E1000_NAPI is not set
CONFIG_NS83820=m
CONFIG_HAMACHI=m
CONFIG_YELLOWFIN=m
CONFIG_R8169=m
# CONFIG_R8169_NAPI is not set
# CONFIG_R8169_VLAN is not set
# CONFIG_SKGE is not set
CONFIG_SK98LIN=m
# CONFIG_VIA_VELOCITY is not set
CONFIG_TIGON3=m
# CONFIG_BNX2 is not set

#
# Ethernet (10000 Mbit)
#
CONFIG_IXGB=m
# CONFIG_IXGB_NAPI is not set
CONFIG_S2IO=m
# CONFIG_S2IO_NAPI is not set
# CONFIG_2BUFF_MODE is not set

#
# Token Ring devices
#
CONFIG_TR=y
CONFIG_IBMTR=m
CONFIG_IBMOL=m
CONFIG_IBMLS=m
CONFIG_3C359=m
CONFIG_TMS380TR=m
CONFIG_TMSPCI=m
CONFIG_SKISA=m
CONFIG_PROTEON=m
CONFIG_ABYSS=m
CONFIG_MADGEMC=m
CONFIG_SMCTR=m

#
# Wireless LAN (non-hamradio)
#
CONFIG_NET_RADIO=y

#
# Obsolete Wireless cards support (pre-802.11)
#
CONFIG_STRIP=m
CONFIG_ARLAN=m
CONFIG_WAVELAN=m

#
# Wireless 802.11b ISA/PCI cards support
#
CONFIG_AIRO=m
CONFIG_HERMES=m
CONFIG_PLX_HERMES=m
CONFIG_TMD_HERMES=m
CONFIG_PCI_HERMES=m
CONFIG_ATMEL=m
CONFIG_PCI_ATMEL=m

#
# Prism GT/Duette 802.11(a/b/g) PCI/Cardbus support
#
CONFIG_PRISM54=m
CONFIG_NET_WIRELESS=y

#
# Wan interfaces
#
CONFIG_WAN=y
CONFIG_HOSTESS_SV11=m
CONFIG_COSA=m
CONFIG_DSCC4=m
CONFIG_DSCC4_PCISYNC=y
# CONFIG_DSCC4_PCI_RST is not set
CONFIG_LANMEDIA=m
CONFIG_SEALEVEL_4021=m
CONFIG_SYNCLINK_SYNCPPP=m
CONFIG_HDLC=m
CONFIG_HDLC_RAW=y
CONFIG_HDLC_RAW_ETH=y
CONFIG_HDLC_CISCO=y
CONFIG_HDLC_FR=y
CONFIG_HDLC_PPP=y
CONFIG_HDLC_X25=y
CONFIG_PCI200SYN=m
CONFIG_WANXL=m
CONFIG_PC300=m
CONFIG_PC300_MLPPP=y
CONFIG_N2=m
CONFIG_C101=m
CONFIG_FARSYNC=m
CONFIG_DLCI=m
CONFIG_DLCI_COUNT=24
CONFIG_DLCI_MAX=8
CONFIG_SDLA=m
CONFIG_WAN_ROUTER_DRIVERS=y
CONFIG_VENDOR_SANGOMA=m
CONFIG_WANPIPE_CHDLC=y
CONFIG_WANPIPE_FR=y
CONFIG_WANPIPE_X25=y
CONFIG_WANPIPE_PPP=y
CONFIG_WANPIPE_MULTPPP=y
CONFIG_CYCLADES_SYNC=m
CONFIG_CYCLOMX_X25=y
CONFIG_LAPBETHER=m
CONFIG_X25_ASY=m
CONFIG_SBNI=m
CONFIG_SBNI_MULTILINE=y

#
# ATM drivers
#
CONFIG_ATM_TCP=m
CONFIG_ATM_LANAI=m
CONFIG_ATM_ENI=m
# CONFIG_ATM_ENI_DEBUG is not set
CONFIG_ATM_ENI_TUNE_BURST=y
# CONFIG_ATM_ENI_BURST_TX_16W is not set
CONFIG_ATM_ENI_BURST_TX_8W=y
CONFIG_ATM_ENI_BURST_TX_4W=y
CONFIG_ATM_ENI_BURST_TX_2W=y
# CONFIG_ATM_ENI_BURST_RX_16W is not set
# CONFIG_ATM_ENI_BURST_RX_8W is not set
CONFIG_ATM_ENI_BURST_RX_4W=y
CONFIG_ATM_ENI_BURST_RX_2W=y
CONFIG_ATM_FIRESTREAM=m
CONFIG_ATM_ZATM=m
# CONFIG_ATM_ZATM_DEBUG is not set
CONFIG_ATM_NICSTAR=m
CONFIG_ATM_NICSTAR_USE_SUNI=y
CONFIG_ATM_NICSTAR_USE_IDT77105=y
CONFIG_ATM_IDT77252=m
# CONFIG_ATM_IDT77252_DEBUG is not set
# CONFIG_ATM_IDT77252_RCV_ALL is not set
CONFIG_ATM_IDT77252_USE_SUNI=y
CONFIG_ATM_AMBASSADOR=m
# CONFIG_ATM_AMBASSADOR_DEBUG is not set
CONFIG_ATM_HORIZON=m
# CONFIG_ATM_HORIZON_DEBUG is not set
CONFIG_ATM_IA=m
# CONFIG_ATM_IA_DEBUG is not set
CONFIG_ATM_FORE200E_MAYBE=m
CONFIG_ATM_FORE200E_PCA=y
CONFIG_ATM_FORE200E_PCA_DEFAULT_FW=y
CONFIG_ATM_FORE200E_USE_TASKLET=y
CONFIG_ATM_FORE200E_TX_RETRY=16
CONFIG_ATM_FORE200E_DEBUG=0
CONFIG_ATM_FORE200E=m
CONFIG_ATM_HE=m
# CONFIG_ATM_HE_USE_SUNI is not set
CONFIG_FDDI=y
CONFIG_DEFXX=m
CONFIG_SKFP=m
CONFIG_HIPPI=y
CONFIG_ROADRUNNER=m
# CONFIG_ROADRUNNER_LARGE_RINGS is not set
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPPOE=m
CONFIG_PPPOATM=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLIP_SMART=y
CONFIG_SLIP_MODE_SLIP6=y
CONFIG_NET_FC=y
CONFIG_SHAPER=m
CONFIG_NETCONSOLE=m
CONFIG_NETPOLL=y
CONFIG_NETPOLL_RX=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_TSDEV=m
CONFIG_INPUT_TSDEV_SCREEN_X=240
CONFIG_INPUT_TSDEV_SCREEN_Y=320
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_SUNKBD=m
CONFIG_KEYBOARD_LKKBD=m
CONFIG_KEYBOARD_XTKBD=m
CONFIG_KEYBOARD_NEWTON=m
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_INPORT=m
# CONFIG_MOUSE_ATIXL is not set
CONFIG_MOUSE_LOGIBM=m
CONFIG_MOUSE_PC110PAD=m
CONFIG_MOUSE_VSXXXAA=m
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
# CONFIG_JOYSTICK_TWIDJOY is not set
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
CONFIG_JOYSTICK_JOYDUMP=m
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_GUNZE=m
# CONFIG_TOUCHSCREEN_ELO is not set
# CONFIG_TOUCHSCREEN_MTOUCH is not set
# CONFIG_TOUCHSCREEN_MK712 is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
CONFIG_INPUT_UINPUT=m

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=m
CONFIG_SERIO_PARKBD=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_COMPUTONE=m
CONFIG_ROCKETPORT=m
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
CONFIG_DIGIEPCA=m
CONFIG_ESPSERIAL=m
CONFIG_MOXA_INTELLIO=m
CONFIG_MOXA_SMARTIO=m
CONFIG_ISI=m
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_N_HDLC=m
CONFIG_RISCOM8=m
CONFIG_SPECIALIX=m
# CONFIG_SPECIALIX_RTSCTS is not set
CONFIG_SX=m
CONFIG_RIO=m
CONFIG_RIO_OLDPCI=y
CONFIG_STALDRV=y
CONFIG_STALLION=m
CONFIG_ISTALLION=m

#
# Serial drivers
#
CONFIG_SERIAL_8250=m
CONFIG_SERIAL_8250_ACPI=y
CONFIG_SERIAL_8250_NR_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set
# CONFIG_SERIAL_8250_MCA is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=m
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_PRINTER=m
CONFIG_LP_CONSOLE=y
CONFIG_PPDEV=m
CONFIG_TIPAR=m

#
# IPMI
#
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
CONFIG_HW_RANDOM=m
CONFIG_NVRAM=m
CONFIG_RTC=m
CONFIG_RTC_HISTOGRAM=m
CONFIG_BLOCKER=y
CONFIG_LPPTEST=m
CONFIG_GEN_RTC=m
CONFIG_GEN_RTC_X=y
CONFIG_DTLK=m
CONFIG_R3964=m
CONFIG_APPLICOM=m
CONFIG_SONYPI=m

#
# Ftape, the floppy tape device driver
#
CONFIG_FTAPE=m
CONFIG_ZFTAPE=m
CONFIG_ZFT_DFLT_BLK_SZ=10240

#
# The compressor will be built as a module only!
#
CONFIG_ZFT_COMPRESSOR=m
CONFIG_FT_NR_BUFFERS=3
CONFIG_FT_PROC_FS=y
CONFIG_FT_NORMAL_DEBUG=y
# CONFIG_FT_FULL_DEBUG is not set
# CONFIG_FT_NO_TRACE is not set
# CONFIG_FT_NO_TRACE_AT_ALL is not set

#
# Hardware configuration
#
CONFIG_FT_STD_FDC=y
# CONFIG_FT_MACH2 is not set
# CONFIG_FT_PROBE_FC10 is not set
# CONFIG_FT_ALT_FDC is not set
CONFIG_FT_FDC_THR=8
CONFIG_FT_FDC_MAX_RATE=2000
CONFIG_FT_ALPHA_CLOCK=0
CONFIG_AGP=m
CONFIG_AGP_ALI=m
CONFIG_AGP_ATI=m
CONFIG_AGP_AMD=m
CONFIG_AGP_AMD64=m
CONFIG_AGP_INTEL=m
CONFIG_AGP_NVIDIA=m
CONFIG_AGP_SIS=m
CONFIG_AGP_SWORKS=m
CONFIG_AGP_VIA=m
CONFIG_AGP_EFFICEON=m
CONFIG_DRM=m
CONFIG_DRM_TDFX=m
# CONFIG_DRM_GAMMA is not set
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_I810=m
CONFIG_DRM_I830=m
CONFIG_DRM_I915=m
CONFIG_DRM_MGA=m
CONFIG_DRM_SIS=m
# CONFIG_DRM_VIA is not set
CONFIG_MWAVE=m
CONFIG_SCx200_GPIO=m
CONFIG_RAW_DRIVER=m
# CONFIG_HPET is not set
CONFIG_MAX_RAW_DEVS=256
CONFIG_HANGCHECK_TIMER=m

#
# TPM devices
#
# CONFIG_TCG_TPM is not set

#
# I2C support
#
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
CONFIG_I2C_ALGOPCA=m

#
# I2C Hardware Bus support
#
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_ELEKTOR=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_I2C=m
CONFIG_SCx200_I2C_SCL=12
CONFIG_SCx200_I2C_SDA=13
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
CONFIG_I2C_STUB=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
CONFIG_I2C_PCA_ISA=m
CONFIG_I2C_SENSOR=m

#
# Miscellaneous I2C Chip support
#
# CONFIG_SENSORS_DS1337 is not set
# CONFIG_SENSORS_DS1374 is not set
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
# CONFIG_SENSORS_PCA9539 is not set
CONFIG_SENSORS_PCF8591=m
# CONFIG_SENSORS_RTC8564 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Hardware Monitoring support
#
CONFIG_HWMON=y
CONFIG_SENSORS_ADM1021=m
# CONFIG_SENSORS_ADM1025 is not set
CONFIG_SENSORS_ADM1026=m
# CONFIG_SENSORS_ADM1031 is not set
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ASB100=m
# CONFIG_SENSORS_ATXP1 is not set
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_FSCPOS=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM75=m
# CONFIG_SENSORS_LM77 is not set
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
# CONFIG_SENSORS_MAX1619 is not set
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Misc devices
#
CONFIG_IBM_ASM=m

#
# Multimedia devices
#
CONFIG_VIDEO_DEV=m

#
# Video For Linux
#

#
# Video Adapters
#
CONFIG_VIDEO_BT848=m
CONFIG_VIDEO_PMS=m
CONFIG_VIDEO_BWQCAM=m
CONFIG_VIDEO_CQCAM=m
CONFIG_VIDEO_W9966=m
CONFIG_VIDEO_CPIA=m
CONFIG_VIDEO_CPIA_PP=m
CONFIG_VIDEO_CPIA_USB=m
CONFIG_VIDEO_SAA5246A=m
CONFIG_VIDEO_SAA5249=m
CONFIG_TUNER_3036=m
CONFIG_VIDEO_STRADIS=m
CONFIG_VIDEO_ZORAN=m
CONFIG_VIDEO_ZORAN_BUZ=m
CONFIG_VIDEO_ZORAN_DC10=m
CONFIG_VIDEO_ZORAN_DC30=m
CONFIG_VIDEO_ZORAN_LML33=m
CONFIG_VIDEO_ZORAN_LML33R10=m
# CONFIG_VIDEO_ZR36120 is not set
CONFIG_VIDEO_MEYE=m
# CONFIG_VIDEO_SAA7134 is not set
CONFIG_VIDEO_MXB=m
CONFIG_VIDEO_DPC=m
CONFIG_VIDEO_HEXIUM_ORION=m
CONFIG_VIDEO_HEXIUM_GEMINI=m
# CONFIG_VIDEO_CX88 is not set
# CONFIG_VIDEO_OVCAMCHIP is not set

#
# Radio Adapters
#
CONFIG_RADIO_CADET=m
CONFIG_RADIO_RTRACK=m
CONFIG_RADIO_RTRACK2=m
CONFIG_RADIO_AZTECH=m
CONFIG_RADIO_GEMTEK=m
CONFIG_RADIO_GEMTEK_PCI=m
CONFIG_RADIO_MAXIRADIO=m
CONFIG_RADIO_MAESTRO=m
CONFIG_RADIO_SF16FMI=m
CONFIG_RADIO_SF16FMR2=m
CONFIG_RADIO_TERRATEC=m
CONFIG_RADIO_TRUST=m
CONFIG_RADIO_TYPHOON=m
CONFIG_RADIO_TYPHOON_PROC_FS=y
CONFIG_RADIO_ZOLTRIX=m

#
# Digital Video Broadcasting Devices
#
CONFIG_DVB=y
CONFIG_DVB_CORE=m

#
# Supported SAA7146 based PCI Adapters
#
CONFIG_DVB_AV7110=m
CONFIG_DVB_AV7110_OSD=y
CONFIG_DVB_BUDGET=m
CONFIG_DVB_BUDGET_CI=m
CONFIG_DVB_BUDGET_AV=m
CONFIG_DVB_BUDGET_PATCH=m

#
# Supported USB Adapters
#
# CONFIG_DVB_USB is not set
CONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_TTUSB_DEC=m
CONFIG_DVB_CINERGYT2=m
# CONFIG_DVB_CINERGYT2_TUNING is not set

#
# Supported FlexCopII (B2C2) Adapters
#
# CONFIG_DVB_B2C2_FLEXCOP is not set

#
# Supported BT878 Adapters
#
CONFIG_DVB_BT8XX=m

#
# Supported Pluto2 Adapters
#
# CONFIG_DVB_PLUTO2 is not set

#
# Supported DVB Frontends
#

#
# Customise DVB Frontends
#

#
# DVB-S (satellite) frontends
#
CONFIG_DVB_STV0299=m
CONFIG_DVB_CX24110=m
CONFIG_DVB_TDA8083=m
CONFIG_DVB_TDA80XX=m
CONFIG_DVB_MT312=m
CONFIG_DVB_VES1X93=m
CONFIG_DVB_S5H1420=m

#
# DVB-T (terrestrial) frontends
#
CONFIG_DVB_SP8870=m
CONFIG_DVB_SP887X=m
CONFIG_DVB_CX22700=m
CONFIG_DVB_CX22702=m
CONFIG_DVB_L64781=m
CONFIG_DVB_TDA1004X=m
CONFIG_DVB_NXT6000=m
CONFIG_DVB_MT352=m
CONFIG_DVB_DIB3000MB=m
CONFIG_DVB_DIB3000MC=m

#
# DVB-C (cable) frontends
#
CONFIG_DVB_ATMEL_AT76C651=m
CONFIG_DVB_VES1820=m
CONFIG_DVB_TDA10021=m
CONFIG_DVB_STV0297=m

#
# ATSC (North American/Korean Terresterial DTV) frontends
#
CONFIG_DVB_NXT2002=m
CONFIG_DVB_OR51211=m
# CONFIG_DVB_OR51132 is not set
# CONFIG_DVB_BCM3510 is not set
# CONFIG_DVB_LGDT330X is not set
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_VIDEO_VIDEOBUF=m
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEO_BUF=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_IR=m
CONFIG_VIDEO_TVEEPROM=m

#
# Graphics support
#
CONFIG_FB=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_SOFT_CURSOR=y
# CONFIG_FB_MACMODES is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
CONFIG_FB_VESA=y
CONFIG_VIDEO_SELECT=y
# CONFIG_FB_HGA is not set
# CONFIG_FB_NVIDIA is not set
CONFIG_FB_RIVA=m
# CONFIG_FB_RIVA_I2C is not set
# CONFIG_FB_RIVA_DEBUG is not set
CONFIG_FB_I810=m
# CONFIG_FB_I810_GTF is not set
CONFIG_FB_INTEL=m
# CONFIG_FB_INTEL_DEBUG is not set
CONFIG_FB_MATROX=m
# CONFIG_FB_MATROX_MILLENIUM is not set
# CONFIG_FB_MATROX_MYSTIQUE is not set
CONFIG_FB_MATROX_G=y
CONFIG_FB_MATROX_I2C=m
# CONFIG_FB_MATROX_MAVEN is not set
CONFIG_FB_MATROX_MULTIHEAD=y
# CONFIG_FB_RADEON_OLD is not set
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
# CONFIG_FB_RADEON_DEBUG is not set
CONFIG_FB_ATY128=m
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
# CONFIG_FB_ATY_GENERIC_LCD is not set
CONFIG_FB_ATY_XL_INIT=y
CONFIG_FB_ATY_GX=y
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
CONFIG_FB_3DFX=m
# CONFIG_FB_3DFX_ACCEL is not set
CONFIG_FB_VOODOO1=m
CONFIG_FB_TRIDENT=m
# CONFIG_FB_TRIDENT_ACCEL is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_VIRTUAL is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_MDA_CONSOLE=m
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y

#
# Logo configuration
#
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_MONO=y
CONFIG_LOGO_LINUX_VGA16=y
CONFIG_LOGO_LINUX_CLUT224=y
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_GENERIC_PM=y

#
# Generic devices
#
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_OPL4_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m

#
# ISA devices
#
CONFIG_SND_AD1848_LIB=m
CONFIG_SND_CS4231_LIB=m
CONFIG_SND_AD1816A=m
CONFIG_SND_AD1848=m
CONFIG_SND_CS4231=m
CONFIG_SND_CS4232=m
CONFIG_SND_CS4236=m
CONFIG_SND_ES968=m
CONFIG_SND_ES1688=m
CONFIG_SND_ES18XX=m
CONFIG_SND_GUS_SYNTH=m
CONFIG_SND_GUSCLASSIC=m
CONFIG_SND_GUSEXTREME=m
CONFIG_SND_GUSMAX=m
CONFIG_SND_INTERWAVE=m
CONFIG_SND_INTERWAVE_STB=m
CONFIG_SND_OPTI92X_AD1848=m
CONFIG_SND_OPTI92X_CS4231=m
CONFIG_SND_OPTI93X=m
CONFIG_SND_SB8=m
CONFIG_SND_SB16=m
CONFIG_SND_SBAWE=m
CONFIG_SND_SB16_CSP=y
CONFIG_SND_WAVEFRONT=m
CONFIG_SND_ALS100=m
CONFIG_SND_AZT2320=m
CONFIG_SND_CMI8330=m
CONFIG_SND_DT019X=m
CONFIG_SND_OPL3SA2=m
CONFIG_SND_SGALAXY=m
CONFIG_SND_SSCAPE=m

#
# PCI devices
#
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_ALI5451=m
CONFIG_SND_ATIIXP=m
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CS4281=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_CA0106=m
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
CONFIG_SND_TRIDENT=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_ALS4000=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_FM801=m
CONFIG_SND_FM801_TEA575X=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
CONFIG_SND_VX222=m
CONFIG_SND_HDA_INTEL=m

#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_SPLIT_ISO=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_ISP116X_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_BIG_ENDIAN is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_SL811_HCD=m

#
# USB Device Class drivers
#
CONFIG_USB_AUDIO=m

#
# USB Bluetooth TTY can only be used with disabled Bluetooth subsystem
#
CONFIG_USB_MIDI=m
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support' may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
# CONFIG_USB_STORAGE_USBAT is not set
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y

#
# USB Input Devices
#
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
CONFIG_HID_FF=y
CONFIG_HID_PID=y
CONFIG_LOGITECH_FF=y
CONFIG_THRUSTMASTER_FF=y
CONFIG_USB_HIDDEV=y

#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
CONFIG_USB_AIPTEK=m
CONFIG_USB_WACOM=m
# CONFIG_USB_ACECAD is not set
CONFIG_USB_KBTAB=m
CONFIG_USB_POWERMATE=m
CONFIG_USB_MTOUCH=m
# CONFIG_USB_ITMTOUCH is not set
# CONFIG_USB_EGALAX is not set
CONFIG_USB_XPAD=m
CONFIG_USB_ATI_REMOTE=m
# CONFIG_USB_KEYSPAN_REMOTE is not set

#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m

#
# USB Multimedia devices
#
CONFIG_USB_DABUSB=m
CONFIG_USB_VICAM=m
CONFIG_USB_DSBR=m
CONFIG_USB_IBMCAM=m
CONFIG_USB_KONICAWC=m
CONFIG_USB_OV511=m
CONFIG_USB_SE401=m
# CONFIG_USB_SN9C102 is not set
CONFIG_USB_STV680=m
CONFIG_USB_PWC=m

#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m

#
# USB Host-to-Host Cables
#
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_GENESYS=y
CONFIG_USB_NET1080=y
CONFIG_USB_PL2301=y
CONFIG_USB_KC2190=y

#
# Intelligent USB Devices/Gadgets
#
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_ZAURUS=y
CONFIG_USB_CDCETHER=y

#
# USB Network Adapters
#
CONFIG_USB_AX8817X=y
# CONFIG_USB_ZD1201 is not set
CONFIG_USB_MON=y

#
# USB port drivers
#
CONFIG_USB_USS720=m

#
# USB Serial Converter support
#
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_AIRPRIME is not set
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
# CONFIG_USB_SERIAL_CP2101 is not set
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KEYSPAN_MPR=y
CONFIG_USB_SERIAL_KEYSPAN_USA28=y
CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y
CONFIG_USB_SERIAL_KEYSPAN_USA19=y
CONFIG_USB_SERIAL_KEYSPAN_USA18X=y
CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y
CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_HP4X is not set
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_EZUSB=y

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_AUERSWALD=m
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
CONFIG_USB_CYTHERM=m
CONFIG_USB_PHIDGETKIT=m
# CONFIG_USB_PHIDGETSERVO is not set
CONFIG_USB_IDMOUSE=m
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
CONFIG_USB_TEST=m

#
# USB DSL modem support
#
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
# CONFIG_USB_CXACRU is not set
# CONFIG_USB_XUSBATM is not set

#
# USB Gadget Support
#
CONFIG_USB_GADGET=m
# CONFIG_USB_GADGET_DEBUG_FILES is not set
CONFIG_USB_GADGET_SELECTED=y
CONFIG_USB_GADGET_NET2280=y
CONFIG_USB_NET2280=m
# CONFIG_USB_GADGET_PXA2XX is not set
# CONFIG_USB_GADGET_GOKU is not set
# CONFIG_USB_GADGET_LH7A40X is not set
# CONFIG_USB_GADGET_OMAP is not set
# CONFIG_USB_GADGET_DUMMY_HCD is not set
CONFIG_USB_GADGET_DUALSPEED=y
CONFIG_USB_ZERO=m
CONFIG_USB_ETH=m
CONFIG_USB_ETH_RNDIS=y
CONFIG_USB_GADGETFS=m
CONFIG_USB_FILE_STORAGE=m
CONFIG_USB_FILE_STORAGE_TEST=y
CONFIG_USB_G_SERIAL=m

#
# MMC/SD Card support
#
CONFIG_MMC=m
# CONFIG_MMC_DEBUG is not set
CONFIG_MMC_BLOCK=m
CONFIG_MMC_WBSD=m

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# SN Devices
#

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=y
CONFIG_JBD_DEBUG=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
# CONFIG_REISERFS_FS_SECURITY is not set
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
# CONFIG_JFS_SECURITY is not set
# CONFIG_JFS_DEBUG is not set
CONFIG_JFS_STATISTICS=y
CONFIG_FS_POSIX_ACL=y

#
# XFS support
#
CONFIG_XFS_FS=m
CONFIG_XFS_EXPORT=y
CONFIG_XFS_RT=y
CONFIG_XFS_QUOTA=y
CONFIG_XFS_SECURITY=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_MINIX_FS=m
CONFIG_ROMFS_FS=m
CONFIG_INOTIFY=y
CONFIG_QUOTA=y
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_DNOTIFY=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVPTS_FS_XATTR is not set
CONFIG_TMPFS=y
CONFIG_TMPFS_XATTR=y
# CONFIG_TMPFS_SECURITY is not set
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_RAMFS=y

#
# Miscellaneous filesystems
#
CONFIG_ADFS_FS=m
CONFIG_ADFS_FS_RW=y
CONFIG_AFFS_FS=m
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_BEFS_FS=m
# CONFIG_BEFS_DEBUG is not set
CONFIG_BFS_FS=m
CONFIG_EFS_FS=m
CONFIG_JFFS_FS=m
CONFIG_JFFS_FS_VERBOSE=0
# CONFIG_JFFS_PROC_FS is not set
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_JFFS2_FS_WRITEBUFFER=y
# CONFIG_JFFS2_COMPRESSION_OPTIONS is not set
CONFIG_JFFS2_ZLIB=y
CONFIG_JFFS2_RTIME=y
# CONFIG_JFFS2_RUBIN is not set
CONFIG_CRAMFS=m
CONFIG_VXFS_FS=m
CONFIG_HPFS_FS=m
CONFIG_QNX4FS_FS=m
CONFIG_QNX4FS_RW=y
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
CONFIG_UFS_FS_WRITE=y

#
# Network File Systems
#
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
# CONFIG_RPCSEC_GSS_SPKM3 is not set
CONFIG_SMB_FS=m
# CONFIG_SMB_NLS_DEFAULT is not set
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_XATTR is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
CONFIG_NCP_FS=m
CONFIG_NCPFS_PACKET_SIGNING=y
CONFIG_NCPFS_IOCTL_LOCKING=y
CONFIG_NCPFS_STRONG=y
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
CONFIG_NCPFS_SMALLDOS=y
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
# CONFIG_CODA_FS_OLD_API is not set
CONFIG_AFS_FS=m
CONFIG_RXRPC=m

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not set
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
# CONFIG_EFI_PARTITION is not set

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m

#
# Profiling support
#
# CONFIG_PROFILING is not set

#
# Kernel hacking
#
CONFIG_PRINTK_TIME=y
# CONFIG_PRINTK_IGNORE_LOGLEVEL is not set
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_IRQ_FLAGS=y
CONFIG_WAKEUP_TIMING=y
# CONFIG_WAKEUP_LATENCY_HIST is not set
CONFIG_PREEMPT_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set
# CONFIG_CRITICAL_IRQSOFF_TIMING is not set
CONFIG_LATENCY_TIMING=y
CONFIG_LATENCY_TRACE=y
CONFIG_MCOUNT=y
CONFIG_RT_DEADLOCK_DETECT=y
# CONFIG_DEBUG_RT_LOCKING_MODE is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_FS is not set
CONFIG_FRAME_POINTER=y
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set

#
# Page alloc debug is incompatible with Software Suspend on i386
#
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y

#
# Security options
#
# CONFIG_KEYS is not set
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_CAPABILITIES=m
CONFIG_SECURITY_ROOTPLUG=m
# CONFIG_SECURITY_SECLVL is not set
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1

#
# Cryptographic options
#
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
# CONFIG_CRYPTO_AES_586 is not set
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
# CONFIG_CRYPTO_TEA is not set
CONFIG_CRYPTO_ARC4=m
# CONFIG_CRYPTO_KHAZAD is not set
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_TEST=m

#
# Hardware crypto devices
#
# CONFIG_CRYPTO_DEV_PADLOCK is not set

#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_PC=y

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 18:23 +0200, Ingo Molnar wrote:
> > 
> > Using IPI Shortcut mode
> > khelper/794[CPU#0]: BUG in set_new_owner at kernel/rt.c:916
> 
> this is a 'must not happen'. Somehow lock->held list got non-empty.  
> Maybe some use-after-free thing? Havent seen it myself.

Well, I added all my patches and the laptop still locked up.

I then used my AMD SMP box config on my laptop and just changed the
processor, and it booted.  Not very well, since I didn't bother to
change any of the other configurations. But it didn't lock up.

So I went back to the laptop's original config, and did one change. I
disabled CONFIG_SCHED_SMT, rebooted, and the system booted up.  It
hasn't locked up after four boots.  It did once get into some crazy bug
with scheduling while atomic, but it just spit out so many of them that
I couldn't see what caused it. I'll turn back on netconsole to see if I
can capture that bug, but it seems to be related to someting different.

So I think something's wrong with the scheduling for hyper threading.

-- Steve

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 20:02 -0400, Steven Rostedt wrote:

> So I went back to the laptop's original config, and did one change. I
> disabled CONFIG_SCHED_SMT, rebooted, and the system booted up.  It
> hasn't locked up after four boots.  It did once get into some crazy bug
> with scheduling while atomic, but it just spit out so many of them that
> I couldn't see what caused it. I'll turn back on netconsole to see if I
> can capture that bug, but it seems to be related to someting different.
> 
> So I think something's wrong with the scheduling for hyper threading.

Spoke too soon.  I added netconsole and booted a couple of more times,
and I got a crash again.  So it seems to happen less often with
CONFIG_SCHED_SMT, (don't know why) but it is not the problem.

Here's the latest dump. May be due to having HR timers configured.  I'll
add my logger and do some more in depth debugging tomorrow.

(the bug happened same place, same time, but not all the time)

-- Steve

Freeing unused kernel memory: 296k freed
softirq-timer/0/4[CPU#0]: BUG in set_new_owner at kernel/rt.c:916
NMI watchdog detected lockup on CPU#1 (50000/50000)

Pid: 14, comm:      softirq-timer/1
EIP: 0060:[<c032ea3f>] CPU: 1
EIP is at __down_mutex+0xdf/0x5d0
 EFLAGS: 00000082    Not tainted  (2.6.13-rc6-rt8)
EAX: c1831884 EBX: 00000000 ECX: 00000000 EDX: cf6bdeec
ESI: cf6ba7c0 EDI: cf6bdefc EBP: c1831870 DS: 007b ES: 007b
CR0: 8005003b CR2: 00000000 CR3: 00474000 CR4: 000006d0
 [<c0141d44>] check_wakeup_timing+0xd4/0x170 (20)
 [<c0141db8>] check_wakeup_timing+0x148/0x170 (12)
 [<c0141c0a>] sub_preempt_count+0x1a/0x20 (64)
 [<c011bd5e>] __wake_up+0x1e/0x80 (12)
 [<c0330813>] _spin_lock_irqsave+0x23/0x60 (12)
 [<c011bd5e>] __wake_up+0x1e/0x80 (8)
 [<c0141a67>] add_preempt_count_ti+0x27/0x100 (4)
 [<c011bd5e>] __wake_up+0x1e/0x80 (12)
 [<c0141b5a>] add_preempt_count+0x1a/0x20 (16)
 [<c012ba65>] run_timer_softirq+0x2b5/0x470 (32)
 [<c0127680>] ksoftirqd+0xf0/0x170 (56)
 [<c0127590>] ksoftirqd+0x0/0x170 (32)
 [<c013979a>] kthread+0xba/0xc0 (4)
 [<c01396e0>] kthread+0x0/0xc0 (28)
 [<c0101385>] kernel_thread_helper+0x5/0x10 (16)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c0141b5a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)

------------------------------
| showing all locks held by: |  (softirq-timer/1/14 [cf6ba7c0,  98]):
------------------------------

NMI watchdog detected lockup on CPU#0 (50000/50000)

Pid: 4, comm:      softirq-timer/0
EIP: 0060:[<c013ea5f>] CPU: 0
EIP is at __down_trylock+0x9f/0x330
 EFLAGS: 00000082    Not tainted  (2.6.13-rc6-rt8)
EAX: c0392238 EBX: 00000000 ECX: 00008000 EDX: 00004773
ESI: c0392224 EDI: cf69c000 EBP: 00000086 DS: 007b ES: 007b
CR0: 8005003b CR2: fff3f000 CR3: 00474000 CR4: 000006d0
 [<c0140b73>] rt_down_trylock+0x33/0x480 (44)
 [<c01218f2>] vprintk+0x162/0x240 (8)
 [<c02092cb>] vscnprintf+0x2b/0x40 (8)
 [<c01218f2>] vprintk+0x162/0x240 (24)
 [<c0141a67>] add_preempt_count_ti+0x27/0x100 (64)
 [<c0141b5a>] add_preempt_count+0x1a/0x20 (28)
 [<c0121787>] printk+0x17/0x20 (20)
 [<c01220c7>] __WARN_ON+0x67/0x90 (12)
 [<c032ec5a>] __down_mutex+0x2fa/0x5d0 (48)
 [<c0331152>] _raw_spin_unlock+0x12/0x30 (100)
 [<c011bd5e>] __wake_up+0x1e/0x80 (8)
 [<c0330813>] _spin_lock_irqsave+0x23/0x60 (12)
 [<c011bd5e>] __wake_up+0x1e/0x80 (8)
 [<c0141a67>] add_preempt_count_ti+0x27/0x100 (4)
 [<c011bd5e>] __wake_up+0x1e/0x80 (12)
 [<c0141b5a>] add_preempt_count+0x1a/0x20 (16)
 [<c012ba65>] run_timer_softirq+0x2b5/0x470 (32)
 [<c0127680>] ksoftirqd+0xf0/0x170 (56)
 [<c0127590>] ksoftirqd+0x0/0x170 (32)
 [<c013979a>] kthread+0xba/0xc0 (4)
 [<c01396e0>] kthread+0x0/0xc0 (28)
 [<c0101385>] kernel_thread_helper+0x5/0x10 (16)
---------------------------
| preempt count: 00000003 ]
| 3-level deep critical section nesting:
----------------------------------------
.. [<c0141b5a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)
.. [<c0141b5a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)
.. [<c0141b5a>] .... add_preempt_count+0x1a/0x20
.....[<00000000>] ..   ( <= stext+0x3feffd68/0x8)

------------------------------
| showing all locks held by: |  (softirq-timer/0/4 [cf697840,  98]):
------------------------------

Re: 2.6.13-rc6-rt3

[Alistair John Strachan]

On Tuesday 16 August 2005 13:18, Ingo Molnar wrote:
> i have released the 2.6.13-rc6-rt3 tree, which can be downloaded from
> the usual place:
>
>   http://redhat.com/~mingo/realtime-preempt/
>
> Changes since 2.6.13-rc6-rt2:

Ingo,

I haven't used any of the RT patches since V0.7.51-xx, but I upgraded to -rt8 
yesterday and had a couple of problems. I've just noticed you released -rt9, 
but I don't think my problem is listed as fixed.. I'll upgrade anyway, in a 
minute.

The problem I'm having is that when the kernel probes my IDE devices it slows 
down, taking ages to complete the probe. Henceforth the kernel seems to work 
at a slower speed doing just about anything (compiling, etc.), but 
interactive performance is okay. It's a bizarre problem.

Of course, I assumed this was due to the latest timer changes, and so I 
disabled CONFIG_HIGH_RES_TIMERS and went back to CONFIG_HPET_TIMER.
This works perfectly.

For the moment, is it worth debugging problems in the high res timers set or 
are there known (fixable) issues?

-- 
Cheers,
Alistair.

'No sense being pessimistic, it probably wouldn't work anyway.'
Third year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: 2.6.13-rc6-rt3

[Thomas Gleixner]

Alistair,

On Thu, 2005-08-18 at 10:57 +0100, Alistair John Strachan wrote:
> The problem I'm having is that when the kernel probes my IDE devices it slows 
> down, taking ages to complete the probe. Henceforth the kernel seems to work 
> at a slower speed doing just about anything (compiling, etc.), but 
> interactive performance is okay. It's a bizarre problem.
> 
> Of course, I assumed this was due to the latest timer changes, and so I 
> disabled CONFIG_HIGH_RES_TIMERS and went back to CONFIG_HPET_TIMER.
> This works perfectly.

Can you please send me your .config which shows the problem ?

tglx

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Wed, 2005-08-17 at 18:23 +0200, Ingo Molnar wrote:

> 
> > And it goes on and on. This happens everytime. Without netconsole, I
> > only get the nonzero lock count error. Also, one of my lockups on SMP
> > had to do with the kernel_thread_helper:
> > 
> > Using IPI Shortcut mode
> > khelper/794[CPU#0]: BUG in set_new_owner at kernel/rt.c:916

This was with netconsole and showed up after a bunch of other bugs. So
this is a side effect of what happened earlier.
> 
> this is a 'must not happen'. Somehow lock->held list got non-empty.  
> Maybe some use-after-free thing? Havent seen it myself.
> 

I started debugging netconsole with the RT patch and found this
happening.  After seeing what's wrong, I looked at the latest git
branch, and it seems to already have a similar solution that I was going
to make. Here's a description of what's wrong.

In net/core/dev.c the following code is in net_rx_action:

		netpoll_poll_lock(dev);

		if (dev->quota <= 0 || dev->poll(dev, &budget)) {
			netpoll_poll_unlock(dev);
			raw_local_irq_disable();
			list_del(&dev->poll_list);
			list_add_tail(&dev->poll_list, &queue->poll_list);
			if (dev->quota < 0)
				dev->quota += dev->weight;
			else
				dev->quota = dev->weight;
		} else {
			netpoll_poll_unlock(dev);

The netpoll_poll_lock and netpoll_poll_unlock look like this (in current RT):

static inline netpoll_poll_lock(struct net_device *dev)
{
	if (dev->npinfo) {
		spin_lock(&dev->npinfo->poll_lock);
		dev->npinfo->poll_owner = smp_processor_id();
	}
}

static inline void netpoll_poll_unlock(struct net_device *dev)
{
	if (dev->npinfo) {
		dev->npinfo->poll_owner = -1;
		spin_unlock(&dev->npinfo->poll_lock);
	}
}


The problem here is that between netpoll_poll_lock and
netpoll_poll_unlock the dev->npinfo gets assigned. So we unlock the
dev->npinfo->poll_lock without ever locking it.

Here's the port from the latest git to solve this. I've CCed the netdev,
since I'm not sure I got all the places for rcu_lock for the netpoll. At
least to solve this problem.  I did boot up the kernel and this patch
did fix my bugs that I was getting using netconsole. (I have one more
patch to send to fix the illegal API messages).

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_ernie/include/linux/netpoll.h
===================================================================
--- linux_realtime_ernie/include/linux/netpoll.h	(revision 296)
+++ linux_realtime_ernie/include/linux/netpoll.h	(working copy)
@@ -60,25 +60,31 @@
 	return ret;
 }
 
-static inline void netpoll_poll_lock(struct net_device *dev)
+static inline void *netpoll_poll_lock(struct net_device *dev)
 {
+	rcu_read_lock();
 	if (dev->npinfo) {
 		spin_lock(&dev->npinfo->poll_lock);
 		dev->npinfo->poll_owner = smp_processor_id();
+		return dev->npinfo;
 	}
+	return NULL;
 }
 
-static inline void netpoll_poll_unlock(struct net_device *dev)
+static inline void netpoll_poll_unlock(void *have)
 {
-	if (dev->npinfo) {
-		dev->npinfo->poll_owner = -1;
-		spin_unlock(&dev->npinfo->poll_lock);
+	struct netpoll_info *npi = have;
+
+	if (npi) {
+		npi->poll_owner = -1;
+		spin_unlock(&npi->poll_lock);
 	}
+	rcu_read_unlock();
 }
 
 #else
 #define netpoll_rx(a) 0
-#define netpoll_poll_lock(a)
+#define netpoll_poll_lock(a) 0
 #define netpoll_poll_unlock(a)
 #endif
 
Index: linux_realtime_ernie/net/core/netpoll.c
===================================================================
--- linux_realtime_ernie/net/core/netpoll.c	(revision 296)
+++ linux_realtime_ernie/net/core/netpoll.c	(working copy)
@@ -726,6 +726,9 @@
 	/* last thing to do is link it to the net device structure */
 	ndev->npinfo = npinfo;
 
+	/* avoid racing with NAPI reading npinfo */
+	synchronize_rcu();
+
 	return 0;
 
  release:
Index: linux_realtime_ernie/net/core/dev.c
===================================================================
--- linux_realtime_ernie/net/core/dev.c	(revision 296)
+++ linux_realtime_ernie/net/core/dev.c	(working copy)
@@ -1723,6 +1723,7 @@
 
 	while (!list_empty(&queue->poll_list)) {
 		struct net_device *dev;
+		void *have;
 
 		if (budget <= 0 || jiffies - start_time > 1)
 			goto softnet_break;
@@ -1735,10 +1736,10 @@
 
 		dev = list_entry(queue->poll_list.next,
 				 struct net_device, poll_list);
-		netpoll_poll_lock(dev);
+		have = netpoll_poll_lock(dev);
 
 		if (dev->quota <= 0 || dev->poll(dev, &budget)) {
-			netpoll_poll_unlock(dev);
+			netpoll_poll_unlock(have);
 			raw_local_irq_disable();
 			list_del(&dev->poll_list);
 			list_add_tail(&dev->poll_list, &queue->poll_list);
@@ -1747,7 +1748,7 @@
 			else
 				dev->quota = dev->weight;
 		} else {
-			netpoll_poll_unlock(dev);
+			netpoll_poll_unlock(have);
 			dev_put(dev);
 			raw_local_irq_disable();
 		}

Re: 2.6.13-rc6-rt6

[Paul E. McKenney]

On Fri, Aug 19, 2005 at 05:22:28PM -0400, Steven Rostedt wrote:
> On Wed, 2005-08-17 at 18:23 +0200, Ingo Molnar wrote:
> 
> > 
> > > And it goes on and on. This happens everytime. Without netconsole, I
> > > only get the nonzero lock count error. Also, one of my lockups on SMP
> > > had to do with the kernel_thread_helper:
> > > 
> > > Using IPI Shortcut mode
> > > khelper/794[CPU#0]: BUG in set_new_owner at kernel/rt.c:916
> 
> This was with netconsole and showed up after a bunch of other bugs. So
> this is a side effect of what happened earlier.
> > 
> > this is a 'must not happen'. Somehow lock->held list got non-empty.  
> > Maybe some use-after-free thing? Havent seen it myself.
> 
> I started debugging netconsole with the RT patch and found this
> happening.  After seeing what's wrong, I looked at the latest git
> branch, and it seems to already have a similar solution that I was going
> to make. Here's a description of what's wrong.
> 
> In net/core/dev.c the following code is in net_rx_action:
> 
> 		netpoll_poll_lock(dev);
> 
> 		if (dev->quota <= 0 || dev->poll(dev, &budget)) {
> 			netpoll_poll_unlock(dev);
> 			raw_local_irq_disable();
> 			list_del(&dev->poll_list);
> 			list_add_tail(&dev->poll_list, &queue->poll_list);
> 			if (dev->quota < 0)
> 				dev->quota += dev->weight;
> 			else
> 				dev->quota = dev->weight;
> 		} else {
> 			netpoll_poll_unlock(dev);
> 
> The netpoll_poll_lock and netpoll_poll_unlock look like this (in current RT):
> 
> static inline netpoll_poll_lock(struct net_device *dev)
> {
> 	if (dev->npinfo) {
> 		spin_lock(&dev->npinfo->poll_lock);
> 		dev->npinfo->poll_owner = smp_processor_id();
> 	}
> }
> 
> static inline void netpoll_poll_unlock(struct net_device *dev)
> {
> 	if (dev->npinfo) {
> 		dev->npinfo->poll_owner = -1;
> 		spin_unlock(&dev->npinfo->poll_lock);
> 	}
> }
> 
> 
> The problem here is that between netpoll_poll_lock and
> netpoll_poll_unlock the dev->npinfo gets assigned. So we unlock the
> dev->npinfo->poll_lock without ever locking it.
> 
> Here's the port from the latest git to solve this. I've CCed the netdev,
> since I'm not sure I got all the places for rcu_lock for the netpoll. At
> least to solve this problem.  I did boot up the kernel and this patch
> did fix my bugs that I was getting using netconsole. (I have one more
> patch to send to fix the illegal API messages).
> 
> -- Steve
> 
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
> 
> Index: linux_realtime_ernie/include/linux/netpoll.h
> ===================================================================
> --- linux_realtime_ernie/include/linux/netpoll.h	(revision 296)
> +++ linux_realtime_ernie/include/linux/netpoll.h	(working copy)
> @@ -60,25 +60,31 @@
>  	return ret;
>  }

Good catch -- but a few changes needed to be perfectly safe:

	static inline void *netpoll_poll_lock(struct net_device *dev)
	{

		struct netpoll_info *npi;

		rcu_read_lock();
		npi = rcu_dereference(dev)->npinfo;
		if (have) {
			spin_lock(&npi->poll_lock);
			npi->poll_owner = smp_processor_id();
			return npi;
		}
		return NULL;
	}

The earlier version could get in trouble if dev->npinfo was set
to NULL while this was executing.

I am assuming that the dev pointer is really what is being RCU-protected,
but this example uses mostly static data structures, so it is hard
for me to tell.  The npinfo pointer is not being RCU protected, as it
appears to never be changed.  The other candidate is the rx_np pointer,
which is set to NULL in netpoll_cleanup.  I suggest a modification
to netpoll_cleanup below that handles both possibilities.  Of course,
I might well be missing something...

> -static inline void netpoll_poll_unlock(struct net_device *dev)
> +static inline void netpoll_poll_unlock(void *have)
>  {
> -	if (dev->npinfo) {
> -		dev->npinfo->poll_owner = -1;
> -		spin_unlock(&dev->npinfo->poll_lock);
> +	struct netpoll_info *npi = have;
> +
> +	if (npi) {
> +		npi->poll_owner = -1;
> +		spin_unlock(&npi->poll_lock);
>  	}
> +	rcu_read_unlock();
>  }
>  
>  #else
>  #define netpoll_rx(a) 0
> -#define netpoll_poll_lock(a)
> +#define netpoll_poll_lock(a) 0
>  #define netpoll_poll_unlock(a)
>  #endif
>  
> Index: linux_realtime_ernie/net/core/netpoll.c
> ===================================================================
> --- linux_realtime_ernie/net/core/netpoll.c	(revision 296)
> +++ linux_realtime_ernie/net/core/netpoll.c	(working copy)

If netpoll_setup() is implicitly tearing down an earlier netpoll_setup(),
then something like Steve's change below might be needed.

> @@ -726,6 +726,9 @@
>  	/* last thing to do is link it to the net device structure */
>  	ndev->npinfo = npinfo;
>  
> +	/* avoid racing with NAPI reading npinfo */
> +	synchronize_rcu();
> +
>  	return 0;
>  
>   release:

Assuming that it is legal to block in netpoll_cleanup(), the following
should work.  The idea is to NULL the dev pointer, wait for all RCU
readers to get done, and only then complete the cleanup.

	void netpoll_cleanup(struct netpoll *np)
	{
		struct netpoll_info *npinfo;
		unsigned long flags;
		struct net_device *dp;

		if (np->dev) {
			dp = np->dev;
			rcu_assign_pointer(np->dev, NULL);
			synchronize_rcu();
			npinfo = dp->npinfo;
			if (npinfo && npinfo->rx_np == np) {
				spin_lock_irqsave(&npinfo->rx_lock, flags);
				npinfo->rx_np = NULL;
				npinfo->rx_flags &= ~NETPOLL_RX_ENABLED;
				spin_unlock_irqrestore(&npinfo->rx_lock, flags);
			}
			dev_put(dp);
		}

	}

Again, I do not fully understand this code, so a grain of salt might
come in handy.  But there definitely need to be some rcu_dereference()
and rcu_assign_pointer() primitives in there somewhere.  ;-)

The following changes look good to me, but, as I said earlier, I do
not claim to fully understand this code.

> Index: linux_realtime_ernie/net/core/dev.c
> ===================================================================
> --- linux_realtime_ernie/net/core/dev.c	(revision 296)
> +++ linux_realtime_ernie/net/core/dev.c	(working copy)
> @@ -1723,6 +1723,7 @@
>  
>  	while (!list_empty(&queue->poll_list)) {
>  		struct net_device *dev;
> +		void *have;
>  
>  		if (budget <= 0 || jiffies - start_time > 1)
>  			goto softnet_break;
> @@ -1735,10 +1736,10 @@
>  
>  		dev = list_entry(queue->poll_list.next,
>  				 struct net_device, poll_list);
> -		netpoll_poll_lock(dev);
> +		have = netpoll_poll_lock(dev);
>  
>  		if (dev->quota <= 0 || dev->poll(dev, &budget)) {
> -			netpoll_poll_unlock(dev);
> +			netpoll_poll_unlock(have);
>  			raw_local_irq_disable();
>  			list_del(&dev->poll_list);
>  			list_add_tail(&dev->poll_list, &queue->poll_list);
> @@ -1747,7 +1748,7 @@
>  			else
>  				dev->quota = dev->weight;
>  		} else {
> -			netpoll_poll_unlock(dev);
> +			netpoll_poll_unlock(have);
>  			dev_put(dev);
>  			raw_local_irq_disable();
>  		}

						Thanx, Paul

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Fri, 2005-08-19 at 15:47 -0700, Paul E. McKenney wrote:

> Good catch -- but a few changes needed to be perfectly safe:
> 
> 	static inline void *netpoll_poll_lock(struct net_device *dev)
> 	{
> 
> 		struct netpoll_info *npi;
> 
> 		rcu_read_lock();
> 		npi = rcu_dereference(dev)->npinfo;
> 		if (have) {

Here I'm sure you mean "if (npi) {" :-)

> 			spin_lock(&npi->poll_lock);
> 			npi->poll_owner = smp_processor_id();
> 			return npi;
> 		}
> 		return NULL;
> 	}
> 
> The earlier version could get in trouble if dev->npinfo was set
> to NULL while this was executing.

Truth be told,  I was just fixing the race with getting the npinfo
pointer set between netpoll_poll_lock and netpoll_poll_unlock.  I wrote
a patch that fixed that but nothing with the rcu_locks.  Then I looked
at the current git tree and saw that they already had my changes, but
also included the rcu locks.  So I just (blindly) added them.


> 
> Again, I do not fully understand this code, so a grain of salt might
> come in handy.  But there definitely need to be some rcu_dereference()
> and rcu_assign_pointer() primitives in there somewhere.  ;-)
> 
> The following changes look good to me, but, as I said earlier, I do
> not claim to fully understand this code.

netpoll has changed quite a bit in the last few releases. I've seen lots
of fixup code sent in (which usually means there's lots of new broken
code ;-)

Anyway, I don't quite fully understand RCU. I read a few of the
documents on your web site, but I haven't had time to really digest it.
Have you taken a look at the latest git tree?  The rcu_locks are used
for net poll quite a bit more there.

-- Steve

Re: 2.6.13-rc6-rt6

[Paul E. McKenney]

On Fri, Aug 19, 2005 at 07:02:42PM -0400, Steven Rostedt wrote:
> On Fri, 2005-08-19 at 15:47 -0700, Paul E. McKenney wrote:
> 
> > Good catch -- but a few changes needed to be perfectly safe:
> > 
> > 	static inline void *netpoll_poll_lock(struct net_device *dev)
> > 	{
> > 
> > 		struct netpoll_info *npi;
> > 
> > 		rcu_read_lock();
> > 		npi = rcu_dereference(dev)->npinfo;
> > 		if (have) {
> 
> Here I'm sure you mean "if (npi) {" :-)

Right you are!  ;-)

> > 			spin_lock(&npi->poll_lock);
> > 			npi->poll_owner = smp_processor_id();
> > 			return npi;
> > 		}
> > 		return NULL;
> > 	}
> > 
> > The earlier version could get in trouble if dev->npinfo was set
> > to NULL while this was executing.
> 
> Truth be told,  I was just fixing the race with getting the npinfo
> pointer set between netpoll_poll_lock and netpoll_poll_unlock.  I wrote
> a patch that fixed that but nothing with the rcu_locks.  Then I looked
> at the current git tree and saw that they already had my changes, but
> also included the rcu locks.  So I just (blindly) added them.

Understood!

> > Again, I do not fully understand this code, so a grain of salt might
> > come in handy.  But there definitely need to be some rcu_dereference()
> > and rcu_assign_pointer() primitives in there somewhere.  ;-)
> > 
> > The following changes look good to me, but, as I said earlier, I do
> > not claim to fully understand this code.
> 
> netpoll has changed quite a bit in the last few releases. I've seen lots
> of fixup code sent in (which usually means there's lots of new broken
> code ;-)
> 
> Anyway, I don't quite fully understand RCU. I read a few of the
> documents on your web site, but I haven't had time to really digest it.
> Have you taken a look at the latest git tree?  The rcu_locks are used
> for net poll quite a bit more there.

Hmmm....  Guess it is time for me to stop procrastinating on better
understanding git...

						Thanx, Paul

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

On Fri, 2005-08-19 at 16:12 -0700, Paul E. McKenney wrote:
> 
> Hmmm....  Guess it is time for me to stop procrastinating on better
> understanding git...

Why? I still don't.  Just go to http://www.kernel.org/ and download the
latest git release (as of now it's -git11).

Of course you need to know the special combination :-)

http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.12.tar.bz2
http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.13-rc6.bz2
http://kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.13-rc6-git11.bz2

-- Steve

Re: 2.6.13-rc6-rt6

[Paul E. McKenney]

On Fri, Aug 19, 2005 at 07:20:20PM -0400, Steven Rostedt wrote:
> On Fri, 2005-08-19 at 16:12 -0700, Paul E. McKenney wrote:
> > 
> > Hmmm....  Guess it is time for me to stop procrastinating on better
> > understanding git...
> 
> Why? I still don't.  Just go to http://www.kernel.org/ and download the
> latest git release (as of now it's -git11).
> 
> Of course you need to know the special combination :-)
> 
> http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.12.tar.bz2
> http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.13-rc6.bz2
> http://kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.13-rc6-git11.bz2

I feel very much enlightened!  ;-)

Thank you very much!!!

							Thanx, Paul

Re: 2.6.13-rc6-rt6

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> Here's the port from the latest git to solve this.
> 
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

thanks, applied.

	Ingo

Re: 2.6.13-rc6-rt6

[Steven Rostedt]

Here's a patch to move the pi_lock out of the "fast path".  Thus, only
threads that need to do PI will need to take it.

   Comments?

Please look for any race conditions or side effects that I might have
missed.

Thanks,

-- Steve

(Ingo and Thomas, this is what I already sent you)

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_goliath/kernel/fork.c
===================================================================
--- linux_realtime_goliath/kernel/fork.c	(revision 300)
+++ linux_realtime_goliath/kernel/fork.c	(working copy)
@@ -1011,6 +1011,7 @@
 	plist_init(&p->pi_waiters, MAX_PRIO);
 	preempt_enable();
 	p->blocked_on = NULL; /* not blocked yet */
+	spin_lock_init(&p->rt_lock);
 
 	p->tgid = p->pid;
 	if (clone_flags & CLONE_THREAD)
Index: linux_realtime_goliath/kernel/rt.c
===================================================================
--- linux_realtime_goliath/kernel/rt.c	(revision 301)
+++ linux_realtime_goliath/kernel/rt.c	(working copy)
@@ -759,6 +759,7 @@
 	 * If the task is blocked on some other task then boost that
 	 * other task (or tasks) too:
 	 */
+	__raw_spin_lock(&p->rt_lock);
 	for (;;) {
 		struct rt_mutex_waiter *w = p->blocked_on;
 #ifdef CONFIG_RT_DEADLOCK_DETECT
@@ -777,7 +778,6 @@
 		TRACE_BUG_ON_LOCKED(!lock);
 		TRACE_BUG_ON_LOCKED(!lock_owner(lock));
 		if (rt_task(p) && plist_empty(&w->pi_list)) {
-			TRACE_BUG_ON_LOCKED(was_rt);
 			plist_init(&w->pi_list, prio);
 			plist_add(&w->pi_list, &lock_owner(lock)->task->pi_waiters);
 
@@ -803,8 +803,10 @@
 
 		pi_walk++;
 
+		__raw_spin_unlock(&p->rt_lock);
 		p = lock_owner(lock)->task;
 		TRACE_BUG_ON_LOCKED(!p);
+		__raw_spin_lock(&p->rt_lock);
 		/*
 		 * If the dependee is already higher-prio then
 		 * no need to boost it, and all further tasks down
@@ -813,6 +815,7 @@
 		if (p->prio <= prio)
 			break;
 	}
+	__raw_spin_unlock(&p->rt_lock);
 }
 
 /*
@@ -869,7 +872,7 @@
 	/* mark the current thread as blocked on the lock */
 	waiter->eip = eip;
 #endif
-	__raw_spin_lock(&pi_lock);
+	__raw_spin_lock(&task->rt_lock);
 	task->blocked_on = waiter;
 	waiter->lock = lock;
 	waiter->ti = ti;
@@ -881,15 +884,23 @@
 	if (!rt_task(task)) {
 		plist_add(&waiter->list, &lock->wait_list);
 		set_lock_owner_pending(lock);
-		__raw_spin_unlock(&pi_lock);
+		__raw_spin_unlock(&task->rt_lock);
 		return;
 	}
 #endif
-	plist_add(&waiter->pi_list, &lock_owner(lock)->task->pi_waiters);
+	__raw_spin_unlock(&task->rt_lock);
+	__raw_spin_lock(&pi_lock);
 	/*
-	 * Add RT tasks to the head:
+	 * We could have been added to the pi list from another task 
+	 * doing a pi_setprio.
 	 */
-	plist_add(&waiter->list, &lock->wait_list);
+	if (likely(plist_empty(&waiter->pi_list))) {
+		plist_add(&waiter->pi_list, &lock_owner(lock)->task->pi_waiters);
+		/*
+		 * Add RT tasks to the head:
+		 */
+		plist_add(&waiter->list, &lock->wait_list);
+	}
 	set_lock_owner_pending(lock);
 	/*
 	 * If the waiter has higher priority than the owner
Index: linux_realtime_goliath/include/linux/init_task.h
===================================================================
--- linux_realtime_goliath/include/linux/init_task.h	(revision 300)
+++ linux_realtime_goliath/include/linux/init_task.h	(working copy)
@@ -112,6 +112,7 @@
 	.proc_lock	= SPIN_LOCK_UNLOCKED(tsk.proc_lock),		\
 	.delayed_put	= LIST_HEAD_INIT(tsk.delayed_put),		\
 	.pi_waiters	= PLIST_INIT(tsk.pi_waiters, MAX_PRIO),		\
+	.rt_lock	= RAW_SPIN_LOCK_UNLOCKED,			\
 	.journal_info	= NULL,						\
 	.cpu_timers	= INIT_CPU_TIMERS(tsk.cpu_timers),		\
 	.fs_excl	= ATOMIC_INIT(0),				\
Index: linux_realtime_goliath/include/linux/sched.h
===================================================================
--- linux_realtime_goliath/include/linux/sched.h	(revision 301)
+++ linux_realtime_goliath/include/linux/sched.h	(working copy)
@@ -950,6 +950,7 @@
 	/* RT deadlock detection and priority inheritance handling */
 	struct rt_mutex_waiter *blocked_on;
 	struct rt_mutex *pending_owner;
+	raw_spinlock_t rt_lock;
 	unsigned long rt_flags;
 
 

[RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Mon, 2005-08-22 at 20:33 +0200, Ingo Molnar wrote:

> any ideas how to get rid of pi_lock altogether?

I've toyed with the idea of adding another raw_spin_lock to the mutex. A
lock specific pi_lock.   Instead of grabbing a global pi_lock, grab the
pi_lock of a lock.  To modify any lock w.r.t PI, you must first grab all
the lock's pi_locks being referenced.

The idea stems from the fact that the kernel must order its taking of
locks to prevent deadlocks.  This way the order of locks that are taken
are also always in order. 

So if you have the following case:

P1 blocked_on L1 owned_by P2 blocked_on L2 owned_by P3 ...

The L1, L2, L3 ... must always be in the same order, otherwise the
kernel itself can have a deadlock.

OK, let me prove this (for myself as well ;-)

Lets go by contradiction.

If we assume that the locks _can_ be in different orders and that no
deadlock would result.

So we could have:

P1 blocked_on L1 owned_by P2 ... blocked on Ln owned_by Pn+1

According to our assumtion, there can exist another list here where

Lx owned_by Px+1 ... blocked_on Ly owned by Py+1 

where y < x <= n  and 1 <= y < x

This means that there exists a process that owns Lx and is blocked on a
lock Ly, and also this means that there's some process that owns Ly is
blocked on Lx. This is a deadlock!

So knowing that this list of locks will always be in the same order,
than there must be someway to capitalize on that and take just the locks
of the locks used instead of a global one.

I haven't fully looked into this idea, but there might be something to
it.  I'm sure it will be complex, and perhaps slow things down on a 2x
system. But it should be scalable, since you are only taking locks that
are being used, and not a global one.

What do you guys think? 

-- Steve

Re: [RFC] RT-patch update to remove the global pi_lock

[Daniel Walker]

On Mon, 2005-08-22 at 15:44 -0400, Steven Rostedt wrote:
> On Mon, 2005-08-22 at 20:33 +0200, Ingo Molnar wrote:
> 
> > any ideas how to get rid of pi_lock altogether?
> 
> I've toyed with the idea of adding another raw_spin_lock to the mutex. A
> lock specific pi_lock.   Instead of grabbing a global pi_lock, grab the
> pi_lock of a lock.  To modify any lock w.r.t PI, you must first grab all
> the lock's pi_locks being referenced.

Are you saying that you want to convert the current system to lock all
the pi_locks for all the locks in the sequence?

It seems like you could make it a per task lock, then only lock the
task's pi_lock for pi operations.

> The idea stems from the fact that the kernel must order its taking of
> locks to prevent deadlocks.  This way the order of locks that are taken
> are also always in order. 
> 
> So if you have the following case:
> 
> P1 blocked_on L1 owned_by P2 blocked_on L2 owned_by P3 ...
> 
> The L1, L2, L3 ... must always be in the same order, otherwise the
> kernel itself can have a deadlock.
> 
> OK, let me prove this (for myself as well ;-)
> 
> Lets go by contradiction.

Proof seems straight forward enough. 

One downside would be an increase in mutex structure size though.

Daniel

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Mon, 2005-08-22 at 15:19 -0700, Daniel Walker wrote:
> On Mon, 2005-08-22 at 15:44 -0400, Steven Rostedt wrote:
> > On Mon, 2005-08-22 at 20:33 +0200, Ingo Molnar wrote:
> > 
> > > any ideas how to get rid of pi_lock altogether?
> > 
> > I've toyed with the idea of adding another raw_spin_lock to the mutex. A
> > lock specific pi_lock.   Instead of grabbing a global pi_lock, grab the
> > pi_lock of a lock.  To modify any lock w.r.t PI, you must first grab all
> > the lock's pi_locks being referenced.
> 
> Are you saying that you want to convert the current system to lock all
> the pi_locks for all the locks in the sequence?
> 
> It seems like you could make it a per task lock, then only lock the
> task's pi_lock for pi operations.

How would you add to a lock with just holding a lock for a task?  When
you are grabbing a lock, you must first grab a raw lock associated to
the lock being grabbed.  Although, I'm starting to look into this idea,
and I'm going to first see if the current wait_lock could suffice.  I
may also need to add an additional lock to the task to follow the lock
-> task -> lock route.  The tasks order should be the same as the locks
when the are bound (holding) a lock. Since the task won't be able to
release it without holding the raw lock of the lock it is releasing.
(boy this gets confusing to talk about, since you need to talk about
locks and the locking method within the lock!)

> 
> > The idea stems from the fact that the kernel must order its taking of
> > locks to prevent deadlocks.  This way the order of locks that are taken
> > are also always in order. 
> > 
> > So if you have the following case:
> > 
> > P1 blocked_on L1 owned_by P2 blocked_on L2 owned_by P3 ...
> > 
> > The L1, L2, L3 ... must always be in the same order, otherwise the
> > kernel itself can have a deadlock.
> > 
> > OK, let me prove this (for myself as well ;-)
> > 
> > Lets go by contradiction.
> 
> Proof seems straight forward enough. 
> 
> One downside would be an increase in mutex structure size though.

If I do need to add an additional lock to the mutex, I would abstract it
all, so that the old global pi_lock can be used if configured.  This
way, a UP or a low memory 2x SMP machine can still use the old method,
but when it needs to grow, switch over to the new non-global pi_locking
method.  But, maybe I can still get away with just using the wait_lock
and not add any more overhead to the size of the mutex.

I've just started to expiremnent with this idea, so I really don't know
yet how this will work.

-- Steve

Re: [RFC] RT-patch update to remove the global pi_lock

[Daniel Walker]

On Mon, 2005-08-22 at 20:26 -0400, Steven Rostedt wrote:

> 
> How would you add to a lock with just holding a lock for a task?  When
> you are grabbing a lock, you must first grab a raw lock associated to
> the lock being grabbed.  Although, I'm starting to look into this idea,
> and I'm going to first see if the current wait_lock could suffice.  I
> may also need to add an additional lock to the task to follow the lock
> -> task -> lock route.  The tasks order should be the same as the locks
> when the are bound (holding) a lock. Since the task won't be able to
> release it without holding the raw lock of the lock it is releasing.
> (boy this gets confusing to talk about, since you need to talk about
> locks and the locking method within the lock!)

You might need to explain that one more time . I'm sure it needs more
though, but the pi_lock just protects another cpu from enter
pi_setprio() . What we really want is to protect only the specific
structures modified inside pi_setprio() . Or that's my understanding .
Are you thinking of something else?

I think you would at least need to lock the wait_lock for each lock that
is looped over inside pi_setprio() . Because you access the wait_list
inside the loop .

There is also a pi_waiters list that is per task. You would need to make
a lock for that, I think . Or protect it somehow .

Daniel

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Mon, 2005-08-22 at 17:51 -0700, Daniel Walker wrote:
> On Mon, 2005-08-22 at 20:26 -0400, Steven Rostedt wrote:
> 
> > 
> > How would you add to a lock with just holding a lock for a task?  When
> > you are grabbing a lock, you must first grab a raw lock associated to
> > the lock being grabbed.  Although, I'm starting to look into this idea,
> > and I'm going to first see if the current wait_lock could suffice.  I
> > may also need to add an additional lock to the task to follow the lock
> > -> task -> lock route.  The tasks order should be the same as the locks
> > when the are bound (holding) a lock. Since the task won't be able to
> > release it without holding the raw lock of the lock it is releasing.
> > (boy this gets confusing to talk about, since you need to talk about
> > locks and the locking method within the lock!)
> 
> You might need to explain that one more time . I'm sure it needs more
> though, but the pi_lock just protects another cpu from enter
> pi_setprio() . What we really want is to protect only the specific
> structures modified inside pi_setprio() . Or that's my understanding .
> Are you thinking of something else?

Nope, nothing else. That's pretty much it.

> 
> I think you would at least need to lock the wait_lock for each lock that
> is looped over inside pi_setprio() . Because you access the wait_list
> inside the loop .

That's what I was thinking. The wait_list locks could be the lock that
is used to replace the pi_lock.

> 
> There is also a pi_waiters list that is per task. You would need to make
> a lock for that, I think . Or protect it somehow .
> 

Yeah this is sort of what I was thinking of when I follow the
lock->task->lock route.

Here's a visual:

X=> : blocked on
->  : owned by

P1 X=> L1 -+
       ^   |
P2 X===+   |
           |
           +->P4 X=> L3 -> P5
           |
P3 X=> L2 -+

So P4 owns locks L1 and L2 with tasks P1 and P2 blocked on L1 and P3
blocked on L2.  And P4 is blocked on lock L3 owned by P5.

This has the following lists:

L1->wait_list => P1 => P2
L2->wait_list => P3
L3->wait_list => P4

P4->pi_waiters => P1 => P2 => P3
P5->pi_waiters => P4


OK, we add another lock to the task that protects the pi_waiters.
Always grab this lock _after_ grabbing all the necessary wait_locks.

So when P5 needs to update it's pi_waiters, it first grabs the
L3->wait_lock, and then the P5->pi_lock.

So when blocking on a lock, you would grab the lock->wait_locks of all
the locks that you are blocked on.  When P2 blocked on a L1 it would
have grabbed (in this order) L1->wait_lock, L3->wait_lock, P4->pi_lock. 

Actually, for simplicity, since the pi_lock is associated to the task,
we may be able to safely grab the task pi_lock before a wait_lock. Say
if we have P5 blocked on L4 owned by P6.  When updating the
P4->pi_waiters (say when P2 blocked on L1), we could do this:

Grab L1->wait_lock, L3->wait_lock, P4->pi_lock, L4->wait_lock.  Since we
should never be grabbing the L4->wait_lock (or higher) when grabbing the
P4->pi_lock.

God, I think a thesis can be made out of this.  Well, let me start
coding, since I'm one of those that write code better than I design.
I'm a Spiral type of guy, not a Waterfall one ;-)
Code crap, write about it, recode it as gold!

-- Steve

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Mon, 2005-08-22 at 21:32 -0400, Steven Rostedt wrote:

> 
> God, I think a thesis can be made out of this.  Well, let me start
> coding, since I'm one of those that write code better than I design.
> I'm a Spiral type of guy, not a Waterfall one ;-)
> Code crap, write about it, recode it as gold!
> 

I'm really made a mess of the code now and having a lot of fun ;-)

I think this can be pulled off, but I'm seeing that the easiest way is
to do the grabbing of locks with lock -> owner -> lock -> owner ...

So if you have the chain.

P1 X=> L1 -> P2 X=> L2 X=> P3

You would always need to grab the locks in this order:

P1->pi_lock, L1->wait_lock, P2->pi_lock, L2->wait_lock, P3->pi_lock

So on a __down, if you don't get the lock, this makes for easy
transition in the pi_setprio.  You have the current->pi_lock, and then
grab the lock->wait_lock that current is blocked on.  In the loop, you
need to first grab p->wait_lock, to prevent the race with p->blocked_on
and setting it up.  Having to get the lock->wait_lock first would mean
there's a race to get blocked_on, and knowing what it was blocked on.

Now the PITA is with the __up.  Here we have the lock, and we need to
change the owner.  So we need to unlock the lock (as well as the current
owner pi_lock), before giving it to the new owner.  So the race is, if
you have the lock->wait_lock, find the new owner, then unlock the
lock->wait_lock, lock new_owner->pi_lock and then grab the
lock->wait_lock again, in this time a higher priority process could have
come and blocked on this lock.  So the new_owner should really be the
high priority process that just came in. This skips by the pending owner
algorithm.

There's more than one solution to solve this.  The easy, inefficient
way, is to have a test to see if this occurred and try again. But I was
thinking of the following.

When grabbing a lock, you check if there are waiters (although there may
not be an owner or even a pending owner), if you are the highest
priority process, grab the lock and go, otherwise, just add yourself to
the waiting list (and perhaps the pi_list).  So when the original owner
giving up the lock finally runs, it won't need to do anymore work.  If
the lock is owned, then it just unlocks everything, and then it's
someone elses problem.  

The situation could even occur that the higher priority process that
came in and took the lock gave it back to the "new" guy, and the lock
isn't owned at all.  So a simple check of, is the "new" guy still
blocked on the lock and the lock is not owned should be good enough to
finish the change owner job.

It's all complex, but what did you expect when removing a global lock
that makes life simple :-)

-- Steve

Re: [RFC] RT-patch update to remove the global pi_lock

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> > One downside would be an increase in mutex structure size though.
> 
> If I do need to add an additional lock to the mutex, I would abstract 
> it all, so that the old global pi_lock can be used if configured.  
> This way, a UP or a low memory 2x SMP machine can still use the old 
> method, but when it needs to grow, switch over to the new non-global 
> pi_locking method.  But, maybe I can still get away with just using 
> the wait_lock and not add any more overhead to the size of the mutex.

We want to reduce configurability, not increase it. Having something 
like a selectable _core design property_ leads to madness and hard to 
maintain code very quickly.

also, we already have a per-lock spinlock. Why not use that?

	Ingo

Re: 2.6.13-rc6-rt6

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> Here's a patch to move the pi_lock out of the "fast path".  Thus, only 
> threads that need to do PI will need to take it.

ok, looks good, applied it. I've renamed p->rt_lock to p->pi_lock. The 
patch gave a 10% wall-clock improvement in hackbench numbers on an 8-way 
box. (This box has pretty high cachemiss costs, so the positive effects 
of such patches show up nicely.)

	Ingo

Re: 2.6.13-rc6-rt8

[Ingo Molnar]

* Peter Bortas <peter@bortas.org> wrote:

> 2.6.13-rc6-rt8 fails to build with my configuration (attached):
> 
> net/built-in.o: In function `ip_rt_init':
> : undefined reference to `__you_cannot_kmalloc_that_much'
> make[1]: *** [.tmp_vmlinux1] Error 1
> make[1]: Leaving directory `/usr/src/linux-2.6.13-rc6'
> make: *** [stamp-build] Error 2

ok, fixed the likely cause of this in -rt12. Could you check whether it 
builds for you now?

	Ingo

Re: [RFC] RT-patch update to remove the global pi_lock

[Thomas Gleixner]

On Wed, 2005-08-24 at 16:56 -0400, Steven Rostedt wrote:

> Also Thomas,
> 
> I'm still triggering that "huh?" statement in pi_setprio, and it is
> always with the softirq-hrtimer thread.  It likes to change its
> priorities, but there's a time when p->normal_prio != normal_prio(p).
> And this is what's giving me a headache.

Yes, the normal_prio() check detects this. I have a look into this.

tglx

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 5534 bytes --]

Well, after turning off hrtimers, I keep getting one bug. A possible
soft lockup with the ext3 code. But this didn't seem to be caused by the
changes I made. So just to be sure, I ran my test on the vanilla
2.6.13-rc6-rt11 and it gave the same bug too.  So, it looks like my
changes are now at par with what is out there with the rt11 release.
They both give the same bug! ;-)

Attached is the test I ran. I did a 

while : ; do ./test3a_rt ; done

Where test3a_rt is a C program that does adding, deleting and reading of
files, by different tasks that are each at a different priority.  Here's
the soft lockup I'm getting:

NFSD: starting 90-second grace period
BUG: test3a_rt:6138, possible softlockup detected on CPU#0!
 [<c01043f3>] dump_stack+0x23/0x30 (20)
 [<c014cf81>] softlockup_detected+0x41/0x60 (24)
 [<c014d090>] softlockup_tick+0xf0/0x100 (24)
 [<c012bbbb>] update_process_times+0x5b/0x80 (28)
 [<c011415e>] smp_apic_timer_interrupt+0xee/0x100 (36)
 [<c0103ec1>] apic_timer_interrupt+0x21/0x28 (124)
 [<c01c1641>] ext3_add_entry+0xd1/0x210 (60)
 [<c01c1e0d>] ext3_add_nondir+0x2d/0x80 (32)
 [<c01c1f1b>] ext3_create+0xbb/0x110 (40)
 [<c0180721>] vfs_create+0xf1/0x170 (44)
 [<c018105b>] open_namei+0x61b/0x6e0 (72)
 [<c016eb5f>] filp_open+0x3f/0x70 (92)
 [<c016f033>] sys_open+0x53/0x100 (44)
 [<c01033da>] syscall_call+0x7/0xb (-8116)
---------------------------
| preempt count: 00010000 ]
| 0-level deep critical section nesting:
----------------------------------------

------------------------------
| showing all locks held by: |  (test3a_rt/6138 [f7ef61a0,  89]):
------------------------------

#001:             [f2eaddac] {(struct semaphore *)(&inode->i_sem)}
... acquired at:               open_namei+0xf9/0x6e0

NMI show regs on CPU#0:

Pid: 6138, comm:            test3a_rt
EIP: 0060:[<c0114929>] CPU: 0
EIP is at nmi_show_all_regs+0x89/0xd0
 EFLAGS: 00000046    Not tainted  (2.6.13-rc6-rt11-real)
EAX: 00000000 EBX: 000017fa ECX: 00000060 EDX: 00000080
ESI: 00000000 EDI: 0006c692 EBP: f07dbd48 DS: 007b ES: 007b
CR0: 8005003b CR2: b7e37ae0 CR3: 30850000 CR4: 000006d0
 [<c01013b4>] show_regs+0x164/0x16c (36)
 [<c0114a98>] nmi_watchdog_tick+0x128/0x2b0 (56)
 [<c01054bf>] default_do_nmi+0x7f/0x190 (52)
 [<c0105630>] do_nmi+0x50/0x60 (24)
 [<c010405a>] nmi_stack_correct+0x1d/0x22 (72)
 [<c014cf86>] softlockup_detected+0x46/0x60 (24)
 [<c014d090>] softlockup_tick+0xf0/0x100 (24)
 [<c012bbbb>] update_process_times+0x5b/0x80 (28)
 [<c011415e>] smp_apic_timer_interrupt+0xee/0x100 (36)
 [<c0103ec1>] apic_timer_interrupt+0x21/0x28 (124)
 [<c01c1641>] ext3_add_entry+0xd1/0x210 (60)
 [<c01c1e0d>] ext3_add_nondir+0x2d/0x80 (32)
 [<c01c1f1b>] ext3_create+0xbb/0x110 (40)
 [<c0180721>] vfs_create+0xf1/0x170 (44)
 [<c018105b>] open_namei+0x61b/0x6e0 (72)
 [<c016eb5f>] filp_open+0x3f/0x70 (92)
 [<c016f033>] sys_open+0x53/0x100 (44)
 [<c01033da>] syscall_call+0x7/0xb (-8116)
---------------------------
| preempt count: 00010001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c0143f5c>] .... add_preempt_count+0x1c/0x20
.....[<c0114a7d>] ..   ( <= nmi_watchdog_tick+0x10d/0x2b0)

------------------------------
| showing all locks held by: |  (test3a_rt/6138 [f7ef61a0,  89]):
------------------------------

#001:             [f2eaddac] {(struct semaphore *)(&inode->i_sem)}
... acquired at:               open_namei+0xf9/0x6e0

NMI show regs on CPU#1:

Pid: 6137, comm:            test3a_rt
EIP: 0060:[<c01cd9c3>] CPU: 1
EIP is at do_get_write_access+0x83/0x560
 EFLAGS: 00000206    Not tainted  (2.6.13-rc6-rt11-real)
EAX: 000e201d EBX: f7ffe000 ECX: f71e49bc EDX: f71e49bc
ESI: f7f6b3bc EDI: f71e49bc EBP: f1409e94 DS: 007b ES: 007b
CR0: 8005003b CR2: b7fdb000 CR3: 326fb000 CR4: 000006d0
 [<c01013b4>] show_regs+0x164/0x16c (36)
 [<c0114a98>] nmi_watchdog_tick+0x128/0x2b0 (56)
 [<c01054bf>] default_do_nmi+0x7f/0x190 (52)
 [<c0105630>] do_nmi+0x50/0x60 (24)
 [<c010405a>] nmi_stack_correct+0x1d/0x22 (132)
 [<c01cded5>] journal_get_write_access+0x35/0x50 (28)
 [<c01c2b02>] ext3_orphan_del+0x1a2/0x230 (80)
 [<c01bae03>] ext3_delete_inode+0x73/0x110 (24)
 [<c018cad9>] generic_delete_inode+0x99/0x160 (28)
 [<c018cd1d>] generic_drop_inode+0x1d/0x30 (12)
 [<c018cda0>] iput+0x70/0x80 (20)
 [<c0181ee0>] sys_unlink+0x110/0x140 (96)
 [<c01033da>] syscall_call+0x7/0xb (-8116)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c0143f5c>] .... add_preempt_count+0x1c/0x20
.....[<c0114a7d>] ..   ( <= nmi_watchdog_tick+0x10d/0x2b0)

------------------------------
| showing all locks held by: |  (test3a_rt/6137 [f5d437f0,  90]):
------------------------------

#001:             [f709cc88] {(struct semaphore *)(&s->s_lock)}
... acquired at:               ext3_orphan_del+0x3a/0x230



Well, looking at this further, it seems to release somehow, and
continue, since I've got multiple "possible soft lockups", by different
threads. But It doesn't seem to go any further.  Somethings spinning
quite a bit, and something else must be starved.  I'll look further into
this.

So, Ingo, what do you think of the changes so far?  Do you feel that it
is stable enough to send you an actual real patch. That way we can work
together in cleaning it up and get all the other kinks out.

Also, let me know if you are getting this.  The last couple of messages
I sent, never reached me on the LKML, and I sent the last one 4 hours
ago. :-(

Thomas, I did get your reply.

-- Steve


[-- Attachment #2: test3a_rt.c --]
[-- Type: text/x-csrc, Size: 10017 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <time.h>
#include <sys/sem.h>
#include <sys/shm.h>
#include <sys/time.h>
#include <sys/wait.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <dirent.h>
#include <fcntl.h>
#include <signal.h>
#include <ctype.h>
#include <sched.h>

#define __USE_GNU
#include <sys/ipc.h>

#ifdef SDR_TOOLS
#include <dump_log.h>
#include <linux/logdev.h>
#include <dump_log.h>
#include <tracer_xml.h>
#include "dump_trace.h"
int logfd = -1;
#else
#define logdev_print(x...) do {} while(0)
#define logdev_switch_set(x) (0)
#endif

#include <list.h>


static char *testdir = "/tmp/test3a";

static int reader (int i);
static int deleter (int i);
static int creator (int i);

typedef int (*callfunc_t)(int);

#define NR_PROCS 5
pid_t pids[NR_PROCS];
callfunc_t dofunc[NR_PROCS] = {
	deleter,
	creator,
	reader,
	reader,
	reader
};

int priorities[NR_PROCS] = {
	9,
	10,
	11,
	12,
	13,
};

key_t semkey;
key_t shmkey;
int semid = -1;
int shmid = -1;
int safe = 0;
int *flags;

static void remove_testdir(void);

void cleanup(void)
{
	int i;

	for (i=0; i < NR_PROCS; i++) {
		if (pids[i])
			kill(pids[i],SIGKILL);
	}

	if (semid >= 0)
		semctl(semid, 0, IPC_RMID);
	if (shmid >= 0)
		shmctl(shmid, IPC_RMID, NULL);
	remove_testdir();

#ifdef SDR_TOOLS
	if (logfd >= 0)
		close_logdev(logfd);
#endif
}

void catchall(int sig)
{
	cleanup();
	psignal(sig,"Caught: ");
	exit(-1);
}

static int compare_timeval(const struct timeval *a, const struct timeval *b)
{
	return (a->tv_sec > b->tv_sec) ? 1 :
		(a->tv_sec < b->tv_sec) ? -1:
		(a->tv_usec > b->tv_usec) ? 1:
		(a->tv_usec < b->tv_usec) ? -1:
		0;
}

static void add_timeval(const struct timeval *a, const struct timeval *b, struct timeval *c)
{

	c->tv_usec = a->tv_usec + b->tv_usec;
	c->tv_sec = a->tv_sec + b->tv_sec;
	while (c->tv_usec > 1000000) {
		c->tv_usec -= 1000000;
		c->tv_sec++;
	}

}

static void sub_timeval(const struct timeval *a, const struct timeval *b, struct timeval *c)
{

	c->tv_usec = a->tv_usec - b->tv_usec;
	c->tv_sec = a->tv_sec - b->tv_sec;
	while (c->tv_usec < 0) {
		c->tv_usec += 1000000;
		c->tv_sec--;
	}

}

struct dir_item {
	struct list_head list;
	char *dir;
};

LIST_HEAD_DECLARE(dirs);

static void read_dirs(char *dirname)
{
	DIR *dir;
	struct dirent *dent;
	struct stat st;
	struct dir_item *item;
	int len = strlen(dirname);
	char *name;
	struct sembuf sops;
	int semid;

	memset(&sops,0,sizeof(sops));
	
	if ((semid = semget(semkey,1,0)) < 0) {
		perror("semget");
		return;
	}
	
	if ((dir = opendir(dirname)) == NULL)
		return;

	while ((dent = readdir(dir))) {
		if (strcmp(dent->d_name,".") == 0 ||
		    strcmp(dent->d_name,"..") == 0)
			continue;

		
		name = malloc(strlen(dent->d_name)+len+2);
		if (!name) {
			goto out;
		}

		strcpy(name,dirname);
		name[len] = '/';
		strcpy(name+len+1,dent->d_name);

		if (stat(name,&st) < 0) {
			perror(name);
			free(name);
			continue;
		}

		if ((S_ISDIR(st.st_mode))) {
			item = malloc(sizeof(*item));
			if (!item) {
				free(name);
				goto out;
			}
			item->dir = name;
			
			list_add_tail(&item->list,&dirs);
		} else {
			free(name);
		}
	}
 out:
	closedir(dir);
}
			
static int reader(int i)
{
	struct sembuf sops;
	int semid;
	struct timeval starttv;
	struct timeval endtv;
	struct timeval tv;
	struct timeval lasttv;
	struct timeval deltatv;
	struct timespec ts = {0,250000000UL};
	time_t t;
	char timebuf[30];
	unsigned long long starttsc, nowtsc;

	memset(&sops,0,sizeof(sops));
	
	if ((semid = semget(semkey,1,0)) < 0) {
		perror("semget");
		return -1;
	}

	if (gettimeofday(&starttv,NULL) < 0) {
		perror("gettimeofday");
		return -1;
	}

	memset (&deltatv,0,sizeof(deltatv));

	endtv.tv_sec = 10;
	endtv.tv_usec = 0;

	add_timeval(&starttv,&endtv,&endtv);
	printf("reader %d grabbing waiting on sem\n",i);
	printf("(id=%d) start time is %ld.%06ld\n",i,starttv.tv_sec,starttv.tv_usec);
	if (semop(semid, &sops, 1) < 0) {
		perror("semop");
		return -1;
	}

	logdev_print(logfd,"reader %d past semaphore\n",i);


	t = time(NULL);
	ctime_r(&t,timebuf);
	timebuf[strlen(timebuf)-1] = 0;

	printf("reader %d starting loop (%s)\n",i,timebuf);
	logdev_print(logfd,"reader %d (pid %d) starting loop (%s)\n",i,getpid(),timebuf);
	lasttv = starttv;
	asm ("rdtsc" : "=A"(starttsc));
	
	chdir("/");
	read_dirs("/tmp");
	
	do {
		struct list_head *p;
		struct dir_item *item;

		asm ("rdtsc" : "=A"(nowtsc));
		if (safe && (nowtsc - starttsc > 8000000000ULL)) {
			printf("now - start > 8000000000\n");
			break;
		}
		
		
		if (list_empty(&dirs)) {
			nanosleep(&ts,NULL);
			read_dirs("/tmp");
			if (list_empty(&dirs))
				/* ?? */
				break;
		}

		p = dirs.next;
		list_del(p);
		item = list_entry(p,struct dir_item, list);
		printf("%d: reading dir %s\n",i,item->dir);
		read_dirs(item->dir);
		free(item->dir);
		free(item);
		
		if (gettimeofday(&tv,NULL) < 0) {
			perror("gettimeofday (in loop)");
			return -1;
		}

		sub_timeval(&tv,&lasttv,&lasttv);
		if (compare_timeval(&deltatv,&lasttv) < 0)
			deltatv = lasttv;
		lasttv = tv;
	} while(compare_timeval(&endtv,&tv) > 0);
	t = time(NULL);
	ctime_r(&t,timebuf);
	timebuf[strlen(timebuf)-1] = 0;
	logdev_print(logfd,"spinner %d (pid %d) done (%s)\n",i,getpid(),timebuf);
	printf("spinner %d ended loop (%s)  (%d.%06d secs delta)\n",
	       i,timebuf,(int)deltatv.tv_sec,(int)deltatv.tv_usec);
	printf(" (id=%d) end time is %ld.%06ld\n",i,tv.tv_sec,tv.tv_usec);
	printf(" (id=%d) end time should be %ld.%06ld\n",i,endtv.tv_sec,endtv.tv_usec);
	

	/*
	 * When the readers are done, stop the others that don't
	 * have any other test to stop with. No locks needed, we
	 * all just set it to one.
	 */
	*flags = 1;
	return 0;
}

void delete_all(char *dirname)
{
	DIR *dir;
	struct dirent *dent;
	int len = strlen(dirname);
	char *name;
	struct sembuf sops;
	int semid;
	
	memset(&sops,0,sizeof(sops));
	
	if ((semid = semget(semkey,1,0)) < 0) {
		perror("semget");
		return;
	}
	
	if ((dir = opendir(dirname)) == NULL)
		return;

	while ((dent = readdir(dir))) {
		if (strcmp(dent->d_name,".") == 0 ||
		    strcmp(dent->d_name,"..") == 0)
			continue;

		
		name = malloc(strlen(dent->d_name)+len+2);
		if (!name) {
			goto out;
		}

		strcpy(name,dirname);
		name[len] = '/';
		strcpy(name+len+1,dent->d_name);

		unlink(name);
		free(name);
	}
 out:
	closedir(dir);
}

static int deleter(int i)
{
	struct sembuf sops;
	int semid;

	memset(&sops,0,sizeof(sops));
	
	if ((semid = semget(semkey,1,0)) < 0) {
		perror("semget");
		return -1;
	}

	printf("deleter %d grabbing waiting on sem\n",i);
	if (semop(semid, &sops, 1) < 0) {
		perror("semop");
		return -1;
	}

	logdev_print(logfd,"deleter %d past semaphore\n",i);

	while (!*flags) {
		delete_all(testdir);
	}

	return 0;
}

static int creator(int i)
{
	struct sembuf sops;
	int semid;
	int x;
	char name[100];

	memset(&sops,0,sizeof(sops));
	
	if ((semid = semget(semkey,1,0)) < 0) {
		perror("semget");
		return -1;
	}

	mkdir(testdir,0777);

	printf("creator %d grabbing waiting on sem\n",i);
	if (semop(semid, &sops, 1) < 0) {
		perror("semop");
		return -1;
	}

	logdev_print(logfd,"creator %d past semaphore\n",i);

	while (!*flags) {
		int fd;
		char *garbage = "garbage\n";
		snprintf(name,100,"%s/dummy%d",testdir,x++);
		if ((fd = open(name,O_CREAT|O_WRONLY,0777)) < 0) {
			perror(name);
			continue;
		}
		write(fd,garbage,strlen(garbage));
		close(fd);
	}

	return 0;
}

static void remove_testdir(void)
{
	delete_all(testdir);
	rmdir(testdir);
}


void usage(char **argv)
{
	char *arg = argv[0];
	char *p = arg+strlen(arg);

	while (p >= arg && *p != '/') p--;
	p++;

	printf("\nusage: %s [-sn]\n"
	       "  -s : safe mode. Have the readers use the tsc to stop\n"
	       "  -n : run without RT\n"
	       "\n",p);
	exit(-1);
}

int main (int argc, char **argv)
{
	int ret=0;
	int i;
	int nr_procs=0;
	int noprio = 0;
	struct sembuf sops;
	int c;

	opterr = 0;
	while ((c=getopt(argc,argv,"hsn")) >= 0) {
		switch (c) {
		case 's':
			safe = 1;
			break;
		case 'n':
			noprio = 1;
			break;
		case 'h':
		default:
			printf("\n");
			if (c != ':' && tolower(optopt) != 'h' && optopt != '?')
				printf("unknown option: %c\n",optopt);
			usage(argv);
		}
	}

	semkey = ftok(argv[0],123);
	shmkey = ftok(argv[0],456);
	
	if ((semid = semget(semkey,1,IPC_CREAT|IPC_EXCL|0600)) < 0) {
		perror("semget");
		exit (-1);
	}

	if ((shmid = shmget(shmkey,30,IPC_CREAT|IPC_EXCL|0600)) < 0) {
		perror("shmget");
		goto out;
	}
	
	if ((flags = shmat(shmid, NULL, 0)) == (void*)-1) {
		perror("shmat");
		goto out;
	}

	*flags = 0;
#ifdef SDR_TOOLS
	if ((logfd = open_logdev(NULL,O_RDWR)) < 0) {
		perror("open_logdev");
		goto out;
	}

	if (logdev_switch_set(1)) {
		perror("logdev_switch_on");
	}
#endif


	/* Grab the semaphore before anyone else can take it. */
	memset(&sops,0,sizeof(sops));
//	sops.sem_flg = SEM_UNDO;
	sops.sem_op = 1;
	if (semop(semid, &sops, 1) < 0) {
		perror("semop");
		ret = -1;
		goto out;
	}

	for (i=0; i < NR_PROCS; i++) {
		struct sched_param p;

		if ((pids[i] = fork()) < 0) {
			perror("fork");
			ret = -1;
			goto out;
		} else if (pids[i] == 0) {
			/* child */
			ret = dofunc[i](i);
			exit(ret);
		}
		nr_procs++;

		if (!noprio && priorities[i]) {
			p.sched_priority = priorities[i];
			if (sched_setscheduler(pids[i],SCHED_FIFO,&p)) {
				perror("sched_setscheduler");
				goto out;
			}
		}

		
		/* parent */
	}

	
	signal(SIGINT,catchall);
	signal(SIGILL,catchall);
	signal(SIGFPE,catchall);
	signal(SIGSEGV,catchall);
	signal(SIGBUS,catchall);

	sleep(1);
	printf("parent zeroing semaphore\n");

	sops.sem_op = -1;
	
	if (semop(semid,&sops,1) < 0) {
		perror("semop");
		ret = -1;
		goto out;
	}

	while (nr_procs) {
		int status;
		pid_t pid;
		if ((pid = wait(&status)) < 0) {
			perror("wait");
			break;
		}
		for (i=0; i < NR_PROCS; i++) {
			if (pids[i] == pid) {
				pids[i] = 0;
				nr_procs--;
			}
		}
		
	}

	if (logdev_switch_set(0)) {
		perror("logdev_switch_off");
	}

 out:
	cleanup();
	exit(ret);
}

[-- Attachment #3: list.h --]
[-- Type: text/x-chdr, Size: 2871 bytes --]

#ifndef __USER_LIST_H
#define __USER_LIST_H

#include <sys/types.h>
#include <unistd.h>

#define DEBUG_LIST

#ifdef DEBUG_LIST
#include <sys/types.h>
#include <signal.h>
#ifndef LIST_BUG
#define LIST_BUG() do { printf(">>>>> list bug <<<< at %d\n",__LINE__); kill(getpid(),SIGSEGV); } while(0)
#endif
#define __LIST_MAGIC 0xdeabbeef
#define __LIST_MAGIC_POISON 0xbeefdead
#define __LIST_MAGIC_COMMA , __LIST_MAGIC
#define __LIST_MAGIC_ASSIGN(x) (x).magic = __LIST_MAGIC;
#else
#define __LIST_MAGIC
#define __LIST_MAGIC_COMMA
#define __LIST_MAGIC_ASSIGN(x) 
#endif

#define __LIST_POISON ((struct list_head *)-1)

struct list_head {
	struct list_head *next;
	struct list_head *prev;
#ifdef DEBUG_LIST
	int magic;
#endif
};

#define LIST_HEAD_INIT(x) { &x, &x __LIST_MAGIC_COMMA}
#define LIST_HEAD_DECLARE(x) struct list_head x = LIST_HEAD_INIT(x)
#define INIT_LIST_HEAD(x) do { (x).next = &x; (x).prev = &x; __LIST_MAGIC_ASSIGN(x)} while (0)

#define list_for_each(p,list) for(p=(list)->next; p != list; p=p->next)
#define list_for_each_reverse(p,list) for(p=(list)->prev; p != list; p=p->prev)
#define list_for_each_safe(p,n,list) for(p=(list)->next,n=p->next; p != list; p=n,n=p->next)

#ifdef DEBUG_LIST
static inline void list_check(struct list_head *x)
{
	struct list_head *p;
	list_for_each_reverse(p,x) { if (p->magic != __LIST_MAGIC) LIST_BUG(); }
	list_for_each(p,x) { if (p->magic != __LIST_MAGIC) LIST_BUG(); }
}
#endif

static inline void list_add(struct list_head *x,struct list_head *list)
{
#ifdef DEBUG_LIST
	if (x->magic == __LIST_MAGIC_POISON && 
	    (x->next != __LIST_POISON || x->prev != __LIST_POISON))
		LIST_BUG();
#endif
	x->next = list->next;
	x->prev = list;
	list->next->prev = x;
	list->next = x;
#ifdef DEBUG_LIST
	x->magic = __LIST_MAGIC;
	list_check(list);
#endif
}

static inline void list_add_tail(struct list_head *x, struct list_head *list)
{
#ifdef DEBUG_LIST
	if (x->magic == __LIST_MAGIC && 
	    (x->next != __LIST_POISON || x->prev != __LIST_POISON))
		LIST_BUG();
#endif
	x->prev = list->prev;
	x->next = list;
	list->prev->next = x;
	list->prev = x;
#ifdef DEBUG_LIST
	x->magic = __LIST_MAGIC;
	list_check(list);
#endif
}

#define list_entry(p,type,list) ((type*)(((char*)(p))-((char*)&(((type*)NULL)->list))))

#define list_empty(list) ((list)->next == (list))
#define list_head(list) ((list)->next)

static inline void list_del(struct list_head *x)
{
#ifdef DEBUG_LIST
	if (x->magic != __LIST_MAGIC)
		LIST_BUG();
	list_check(x);
	x->magic = __LIST_MAGIC_POISON;
#endif
	x->prev->next = x->next;
	x->next->prev = x->prev;
	x->next = __LIST_POISON;
	x->prev = __LIST_POISON;
}

static inline struct list_head * pop_list_head(struct list_head *head)
{
	struct list_head *p;
	if (list_empty(head))
		return NULL;
	p = head->next;
	list_del(p);
#ifdef DEBUG_LIST
	list_check(head);
#endif
	return p;
}

#endif /* __USER_LIST_H */

Re: [RFC] RT-patch update to remove the global pi_lock

[Daniel Walker]

On Wed, 2005-08-24 at 21:13 -0400, Steven Rostedt wrote:
> Well, after turning off hrtimers, I keep getting one bug. A possible
> soft lockup with the ext3 code. But this didn't seem to be caused by the
> changes I made. So just to be sure, I ran my test on the vanilla
> 2.6.13-rc6-rt11 and it gave the same bug too.  So, it looks like my
> changes are now at par with what is out there with the rt11 release.
> They both give the same bug! ;-)
> 
> Attached is the test I ran. I did a 
> 
> while : ; do ./test3a_rt ; done
> 
> Where test3a_rt is a C program that does adding, deleting and reading of
> files, by different tasks that are each at a different priority.  Here's
> the soft lockup I'm getting:

I got a report of a possible softlockup with setiathome, the trace
wasn't a little garbled though . I'm not sure the checking is working
correctly . But if it is maybe these spot need some performance
analysis . Didn't you changes catch anything that stays in the kernel
for 10 seconds or more ?

Daniel

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Wed, 24 Aug 2005, Daniel Walker wrote:

>
> I got a report of a possible softlockup with setiathome, the trace
> wasn't a little garbled though . I'm not sure the checking is working
> correctly . But if it is maybe these spot need some performance
> analysis . Didn't you changes catch anything that stays in the kernel
> for 10 seconds or more ?
>

I haven't looked that hard yet.  If Ingo is ready for my changes, to
replace the pi_lock, then I can spend time tracking this down.

Something is definity wrong, regardless whether or not the output is
right.  The tests constantly spit crap out, but when this happens,
everything is silent.  So something is definity amiss.

Try it out and see what I mean.

-- Steve

Re: [RFC] RT-patch update to remove the global pi_lock

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> So, Ingo, what do you think of the changes so far?  Do you feel that 
> it is stable enough to send you an actual real patch. That way we can 
> work together in cleaning it up and get all the other kinks out.

yeah, please send me a patch against 2.6.13-rc7-rt1 if possible.

	Ingo

Re: [RFC] RT-patch update to remove the global pi_lock

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> Well, after turning off hrtimers, I keep getting one bug. A possible 
> soft lockup with the ext3 code. But this didn't seem to be caused by 
> the changes I made. So just to be sure, I ran my test on the vanilla 
> 2.6.13-rc6-rt11 and it gave the same bug too.  So, it looks like my 
> changes are now at par with what is out there with the rt11 release. 
> They both give the same bug! ;-)

does the system truly lock up, or is this some transitional condition?  
In any case, i agree that this should be debugged independently of the 
pi_lock patch.

	Ingo

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

OK, here it is. Against 2.6.13-rc7-rt1. 

Please, test it well. Make sure to turn off any debugging, especially
CONFIG_RT_DEADLOCK_DETECT.  Since that would use a global trace_lock
that defeats the purpose of removing the pi_lock.

The patch to remove the single pi_lock:

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_goliath/kernel/rt.c
===================================================================
--- linux_realtime_goliath/kernel/rt.c	(revision 304)
+++ linux_realtime_goliath/kernel/rt.c	(working copy)
@@ -22,6 +22,9 @@
  *
  *  Copyright (C) 2005, Kihon Technologies Inc., Steven Rostedt
  *
+ *   (also by Steven Rostedt)
+ *    - Converted single pi_lock to individual task locks.
+ *
  */
 #include <linux/config.h>
 #include <linux/sched.h>
@@ -36,6 +39,26 @@
 #define CAPTURE_LOCK
 
 /*
+ * Lock order:
+ *
+ *  To keep from having a single lock for PI, each task and lock
+ *  has their own locking. The order is as follows:
+ *
+ * blocked task->pi_lock -> lock->wait_lock -> owner task->pi_lock.
+ * 
+ * This is safe since a owner task should never block on a lock that
+ * is owned by a blocking task.  Otherwise you would have a deadlock 
+ * in the normal system.
+ * The same goes for the locks. A lock held by one task, should not be
+ * taken by task that holds a lock that is blocking this lock's owner.
+ *
+ * A task that is about to grab a lock is first considered to be a 
+ * blocking task, even if the task successfully acquires the lock.
+ * This is because the taking of the locks happen before the
+ * task becomes the owner.
+ */
+
+/*
  * These flags are used for allowing of stealing of ownerships.
  */
 #define RT_PENDOWNER	1	/* pending owner on a lock */
@@ -83,14 +106,6 @@
  */
 //#define ALL_TASKS_PI
 
-/*
- * We need a global lock for priority inheritance handling.
- * This is only for the slow path, but still, we might want
- * to optimize it later to be more scalable.
- */
-static __cacheline_aligned_in_smp raw_spinlock_t pi_lock =
-						RAW_SPIN_LOCK_UNLOCKED;
-
 #ifdef CONFIG_RT_DEADLOCK_DETECT
 # define __EIP_DECL__ , unsigned long eip
 # define __EIP__ , eip
@@ -174,6 +189,8 @@
 	if (trace_on) {				\
 		trace_on = 0;			\
 		console_verbose();		\
+		if (spin_is_locked(&current->pi_lock)) \
+			__raw_spin_unlock(&current->pi_lock); \
 		spin_unlock(&trace_lock);	\
 	}					\
 } while (0)
@@ -228,7 +245,6 @@
  */
 void zap_rt_locks(void)
 {
-	spin_lock_init(&pi_lock);
 #ifdef CONFIG_RT_DEADLOCK_DETECT
 	spin_lock_init(&trace_lock);
 #endif
@@ -589,11 +605,11 @@
 			printk("exiting task is not even the owner??\n");
 		goto restart;
 	}
-	__raw_spin_lock(&pi_lock);
+	__raw_spin_lock(&task->pi_lock);
 	plist_for_each(curr1, &task->pi_waiters) {
 		w = plist_entry(curr1, struct rt_mutex_waiter, pi_list);
 		TRACE_OFF();
-		__raw_spin_unlock(&pi_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 		printk("hm, PI interest held at exit time? Task:\n");
@@ -601,7 +617,7 @@
 		printk_waiter(w);
 		return;
 	}
-	__raw_spin_unlock(&pi_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 }
 
@@ -674,6 +690,7 @@
 	struct rt_mutex_waiter *w;
 	struct plist *curr1;
 
+	__raw_spin_lock(&old_owner->task->pi_lock);
 	plist_for_each(curr1, &old_owner->task->pi_waiters) {
 		w = plist_entry(curr1, struct rt_mutex_waiter, pi_list);
 		if (w->lock == lock) {
@@ -682,9 +699,10 @@
 			printk_waiter(w);
 			printk("\n");
 			TRACE_WARN_ON(1);
-			return;
+			break;
 		}
 	}
+	__raw_spin_unlock(&old_owner->task->pi_lock);
 }
 
 #else
@@ -732,13 +750,24 @@
 
 int pi_walk, pi_null, pi_prio, pi_initialized;
 
-static void pi_setprio(struct rt_mutex *lock, struct task_struct *p, int prio)
+/*
+ * The lock->wait_lock and p->pi_lock must be held.
+ */
+static void pi_setprio(struct rt_mutex *lock, struct task_struct *task, int prio)
 {
+	struct rt_mutex *l = lock;
+	struct task_struct *p = task;
+	/*
+	 * We don't want to release the parameters locks.
+	 */
+
 	if (unlikely(!p->pid)) {
 		pi_null++;
 		return;
 	}
 
+	TRACE_BUG_ON_LOCKED(!spin_is_locked(&lock->wait_lock));
+	TRACE_BUG_ON_LOCKED(!spin_is_locked(&p->pi_lock));
 #ifdef CONFIG_RT_DEADLOCK_DETECT
 	pi_prio++;
 	if (p->policy != SCHED_NORMAL && prio > normal_prio(p)) {
@@ -759,7 +788,6 @@
 	 * If the task is blocked on some other task then boost that
 	 * other task (or tasks) too:
 	 */
-	__raw_spin_lock(&p->pi_lock);
 	for (;;) {
 		struct rt_mutex_waiter *w = p->blocked_on;
 #ifdef CONFIG_RT_DEADLOCK_DETECT
@@ -774,16 +802,27 @@
 		 * it RT, then register the task in the PI list and
 		 * requeue it to the wait list:
 		 */
-		lock = w->lock;
+
+		/*
+		 * Don't unlock the original lock->wait_lock
+		 */
+		if (l != lock)
+			__raw_spin_unlock(&l->wait_lock);
+		l = w->lock;
 		TRACE_BUG_ON_LOCKED(!lock);
-		TRACE_BUG_ON_LOCKED(!lock_owner(lock));
+
+		if (l != lock)
+			__raw_spin_lock(&l->wait_lock);
+
+		TRACE_BUG_ON_LOCKED(!lock_owner(l));
 		if (rt_task(p) && plist_empty(&w->pi_list)) {
+			TRACE_BUG_ON_LOCKED(was_rt);
 			plist_init(&w->pi_list, prio);
-			plist_add(&w->pi_list, &lock_owner(lock)->task->pi_waiters);
+			plist_add(&w->pi_list, &lock_owner(l)->task->pi_waiters);
 
-			plist_del(&w->list, &lock->wait_list);
+			plist_del(&w->list, &l->wait_list);
 			plist_init(&w->list, prio);
-			plist_add(&w->list, &lock->wait_list);
+			plist_add(&w->list, &l->wait_list);
 		}
 		/*
 		 * If the task is blocked on a lock, and we just restored
@@ -795,16 +834,18 @@
 		 */
 		if (!rt_task(p) && !plist_empty(&w->pi_list)) {
 			TRACE_BUG_ON_LOCKED(!was_rt);
-			plist_del(&w->pi_list, &lock_owner(lock)->task->pi_waiters);
-			plist_del(&w->list, &lock->wait_list);
+			plist_del(&w->pi_list, &lock_owner(l)->task->pi_waiters);
+			plist_del(&w->list, &l->wait_list);
 			plist_init(&w->list, prio);
-			plist_add(&w->list, &lock->wait_list);
+			plist_add(&w->list, &l->wait_list);
 		}
 
 		pi_walk++;
+		
+		if (p != task)
+			__raw_spin_unlock(&p->pi_lock);
 
-		__raw_spin_unlock(&p->pi_lock);
-		p = lock_owner(lock)->task;
+		p = lock_owner(l)->task;
 		TRACE_BUG_ON_LOCKED(!p);
 		__raw_spin_lock(&p->pi_lock);
 		/*
@@ -815,7 +856,10 @@
 		if (p->prio <= prio)
 			break;
 	}
-	__raw_spin_unlock(&p->pi_lock);
+	if (l != lock)
+		__raw_spin_unlock(&l->wait_lock);
+	if (p != task)
+		__raw_spin_unlock(&p->pi_lock);
 }
 
 /*
@@ -831,7 +875,9 @@
 	unsigned long flags;
 	int oldprio;
 
-	spin_lock_irqsave(&pi_lock, flags);
+	spin_lock_irqsave(&p->pi_lock,flags);
+	if (p->blocked_on)
+		spin_lock(&p->blocked_on->lock->wait_lock);
 
 	oldprio = p->normal_prio;
 	if (oldprio == prio)
@@ -858,10 +904,17 @@
 	else
 		mutex_setprio(p, prio);
  out:
-	spin_unlock_irqrestore(&pi_lock, flags);
+	if (p->blocked_on)
+		spin_unlock(&p->blocked_on->lock->wait_lock);
 
+	spin_unlock_irqrestore(&p->pi_lock, flags);
+
 }
 
+/*
+ * This is called with both the waiter->task->pi_lock and 
+ * lock->wait_lock held.
+ */
 static void
 task_blocks_on_lock(struct rt_mutex_waiter *waiter, struct thread_info *ti,
 		    struct rt_mutex *lock __EIP_DECL__)
@@ -872,7 +925,6 @@
 	/* mark the current thread as blocked on the lock */
 	waiter->eip = eip;
 #endif
-	__raw_spin_lock(&task->pi_lock);
 	task->blocked_on = waiter;
 	waiter->lock = lock;
 	waiter->ti = ti;
@@ -884,31 +936,24 @@
 	if (!rt_task(task)) {
 		plist_add(&waiter->list, &lock->wait_list);
 		set_lock_owner_pending(lock);
-		__raw_spin_unlock(&task->pi_lock);
 		return;
 	}
 #endif
-	__raw_spin_unlock(&task->pi_lock);
-	__raw_spin_lock(&pi_lock);
+	plist_add(&waiter->pi_list, &lock_owner(lock)->task->pi_waiters);
 	/*
-	 * We could have been added to the pi list from another task
-	 * doing a pi_setprio.
+	 * Add RT tasks to the head:
 	 */
-	if (likely(plist_empty(&waiter->pi_list))) {
-		plist_add(&waiter->pi_list, &lock_owner(lock)->task->pi_waiters);
-		/*
-		 * Add RT tasks to the head:
-		 */
-		plist_add(&waiter->list, &lock->wait_list);
-	}
+	plist_add(&waiter->list, &lock->wait_list);
 	set_lock_owner_pending(lock);
 	/*
 	 * If the waiter has higher priority than the owner
 	 * then temporarily boost the owner:
 	 */
-	if (task->prio < lock_owner(lock)->task->prio)
+	if (task->prio < lock_owner(lock)->task->prio) {
+		__raw_spin_lock(&lock_owner(lock)->task->pi_lock);
 		pi_setprio(lock, lock_owner(lock)->task, task->prio);
-	__raw_spin_unlock(&pi_lock);
+		__raw_spin_unlock(&lock_owner(lock)->task->pi_lock);
+	}
 }
 
 /*
@@ -942,6 +987,10 @@
 }
 EXPORT_SYMBOL(__init_rwsem);
 
+/*
+ * This must be called with both the old_owner and new_owner pi_locks held.
+ * As well as the lock->wait_lock.
+ */
 static inline
 void set_new_owner(struct rt_mutex *lock, struct thread_info *old_owner,
 			struct thread_info *new_owner __EIP_DECL__)
@@ -975,6 +1024,8 @@
 /*
  * handle the lock release when processes blocked on it that can now run
  * - the spinlock must be held by the caller
+ *
+ * The lock->wait_lock must be held, and the lock's owner->pi_lock must not.
  */
 static struct thread_info *
 pick_new_owner(struct rt_mutex *lock, struct thread_info *old_owner,
@@ -990,14 +1041,34 @@
 	 */
 	waiter = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, list);
 
+ try_again:
 	trace_special_pid(waiter->ti->task->pid, waiter->ti->task->prio, 0);
 
 #ifdef ALL_TASKS_PI
 	check_pi_list_present(lock, waiter, old_owner);
 #endif
 	new_owner = waiter->ti;
+	/*
+	 * The new owner is still blocked on this lock, so we
+	 * must release the lock->wait_lock before grabing 
+	 * the new_owner lock.
+	 */
+	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_lock(&new_owner->task->pi_lock);
+	__raw_spin_lock(&lock->wait_lock);
+	/*
+	 * In this split second of releasing the lock, a high priority 
+	 * process could have come along and blocked as well.
+	 */
+	waiter = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, list);
+	if (unlikely(waiter->ti != new_owner)) {
+		__raw_spin_unlock(&new_owner->task->pi_lock);
+		goto try_again;
+	}
 	plist_del_init(&waiter->list, &lock->wait_list);
 
+	__raw_spin_lock(&old_owner->task->pi_lock);
+
 	plist_del(&waiter->pi_list, &old_owner->task->pi_waiters);
 	plist_init(&waiter->pi_list, waiter->ti->task->prio);
 
@@ -1008,6 +1079,9 @@
 	new_owner->task->blocked_on = NULL;
 	TRACE_WARN_ON(save_state != lock->save_state);
 
+	__raw_spin_unlock(&old_owner->task->pi_lock);
+	__raw_spin_unlock(&new_owner->task->pi_lock);
+
 	return new_owner;
 }
 
@@ -1060,9 +1134,7 @@
 	 * is still sleeping and hasn't woken up to get the lock.
 	 */
 
-	/* Test the simple case first, is it already running? */
-	if (!TASK_PENDING(owner))
-		return 0;
+	TRACE_BUG_ON_LOCKED(!owner);
 
 	/* The owner is pending on a lock, but is it this lock? */
 	if (owner->pending_owner != lock)
@@ -1134,17 +1206,35 @@
 		return 0;
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
+	/*
+	 * We are no longer blocked on the lock, so we are considered a
+	 * owner. So we must grab the lock->wait_lock first.
+	 */
 	__raw_spin_lock(&lock->wait_lock);
+	__raw_spin_lock(&task->pi_lock);
 
 	if (!(task->rt_flags & RT_PENDOWNER)) {
-		/* someone else stole it */
+		/*
+		 * Someone else stole it.
+		 * We are grabbing a lock now, so we must release the
+		 * locks again and retake them in the opposite order.
+		 */
+		__raw_spin_unlock(&task->pi_lock);
+		__raw_spin_unlock(&lock->wait_lock);
+
+		__raw_spin_lock(&task->pi_lock);
+		__raw_spin_lock(&lock->wait_lock);
+
 		old_owner = lock_owner(lock);
 		TRACE_BUG_ON_LOCKED(old_owner == ti);
 		if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 			/* we got it back! */
-			__raw_spin_lock(&pi_lock);
-			set_new_owner(lock, old_owner, ti __W_EIP__(waiter));
-			__raw_spin_unlock(&pi_lock);
+			if (old_owner) {
+				__raw_spin_lock(&old_owner->task->pi_lock);
+				set_new_owner(lock, old_owner, ti __W_EIP__(waiter));
+				__raw_spin_unlock(&old_owner->task->pi_lock);
+			} else
+				set_new_owner(lock, old_owner, ti __W_EIP__(waiter));
 			ret = 0;
 		} else {
 			/* Add ourselves back to the list */
@@ -1159,6 +1249,7 @@
 	}
 
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 	return ret;
@@ -1196,6 +1287,7 @@
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
+	__raw_spin_lock(&task->pi_lock);
 	__raw_spin_lock(&lock->wait_lock);
 	INIT_WAITER(&waiter);
 
@@ -1205,10 +1297,14 @@
 	if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 		/* granted */
 		TRACE_WARN_ON_LOCKED(!plist_empty(&lock->wait_list) && !old_owner);
-		__raw_spin_lock(&pi_lock);
-		set_new_owner(lock, old_owner, ti __EIP__);
-		__raw_spin_unlock(&pi_lock);
+		if (old_owner) {
+			__raw_spin_lock(&old_owner->task->pi_lock);
+			set_new_owner(lock, old_owner, ti __EIP__);
+			__raw_spin_unlock(&old_owner->task->pi_lock);
+		} else
+			set_new_owner(lock, old_owner, ti __EIP__);
 		__raw_spin_unlock(&lock->wait_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 		FREE_WAITER(&waiter);
@@ -1223,6 +1319,7 @@
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
 	/* we don't need to touch the lock struct anymore */
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 	might_sleep();
@@ -1273,6 +1370,7 @@
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
+	__raw_spin_lock(&task->pi_lock);
 	__raw_spin_lock(&lock->wait_lock);
 	INIT_WAITER(&waiter);
 
@@ -1282,10 +1380,14 @@
 	if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 		/* granted */
 		TRACE_WARN_ON_LOCKED(!plist_empty(&lock->wait_list) && !old_owner);
-		__raw_spin_lock(&pi_lock);
-		set_new_owner(lock, old_owner, ti __EIP__);
-		__raw_spin_unlock(&pi_lock);
+		if (old_owner) {
+			__raw_spin_lock(&old_owner->task->pi_lock);
+			set_new_owner(lock, old_owner, ti __EIP__);
+			__raw_spin_unlock(&old_owner->task->pi_lock);
+		} else
+			set_new_owner(lock, old_owner, ti __EIP__);
 		__raw_spin_unlock(&lock->wait_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 		FREE_WAITER(&waiter);
@@ -1306,6 +1408,7 @@
 
 	/* we don't need to touch the lock struct anymore */
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock(&trace_lock, ti);
 
 	/*
@@ -1396,7 +1499,6 @@
 	TRACE_WARN_ON_LOCKED(list_empty(&lock->held_list));
 	list_del_init(&lock->held_list);
 #endif
-	__raw_spin_lock(&pi_lock);
 
 #ifdef ALL_TASKS_PI
 	if (plist_empty(&lock->wait_list))
@@ -1409,7 +1511,6 @@
 			__up_mutex_waiter_nosavestate(lock __EIP__);
 	} else
 		lock->owner = NULL;
-	__raw_spin_unlock(&pi_lock);
 	__raw_spin_unlock(&lock->wait_lock);
 #ifdef CONFIG_DEBUG_PREEMPT
 	if (current->lock_count < 0 || current->lock_count >= MAX_LOCK_STACK) {
@@ -1640,6 +1741,7 @@
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
+	__raw_spin_lock(&task->pi_lock);
 	__raw_spin_lock(&lock->wait_lock);
 	INIT_WAITER(&waiter);
 
@@ -1649,10 +1751,14 @@
 	if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 		/* granted */
 		TRACE_WARN_ON_LOCKED(!plist_empty(&lock->wait_list) && !old_owner);
-		__raw_spin_lock(&pi_lock);
-		set_new_owner(lock, old_owner, ti __EIP__);
-		__raw_spin_unlock(&pi_lock);
+		if (old_owner) {
+			__raw_spin_lock(&old_owner->task->pi_lock);
+			set_new_owner(lock, old_owner, ti __EIP__);
+			__raw_spin_unlock(&old_owner->task->pi_lock);
+		} else
+			set_new_owner(lock, old_owner, ti __EIP__);
 		__raw_spin_unlock(&lock->wait_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 		FREE_WAITER(&waiter);
@@ -1667,6 +1773,7 @@
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
 	/* we don't need to touch the lock struct anymore */
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 	might_sleep();
@@ -1684,6 +1791,7 @@
 			 * didnt get the lock - else return success:
 			 */
 			trace_lock_irq(&trace_lock, ti);
+			__raw_spin_lock(&task->pi_lock);
 			__raw_spin_lock(&lock->wait_lock);
 			if (waiter.ti) {
 				plist_del_init(&waiter.list, &lock->wait_list);
@@ -1692,17 +1800,16 @@
 				 * (No big problem if our PI effect lingers
 				 *  a bit - owner will restore prio.)
 				 */
-				__raw_spin_lock(&pi_lock);
 				TRACE_WARN_ON_LOCKED(waiter.ti != ti);
 				TRACE_WARN_ON_LOCKED(current->blocked_on != &waiter);
 				plist_del(&waiter.pi_list, &task->pi_waiters);
 				plist_init(&waiter.pi_list, task->prio);
 				waiter.ti = NULL;
 				current->blocked_on = NULL;
-				__raw_spin_unlock(&pi_lock);
 				ret = -EINTR;
 			}
 			__raw_spin_unlock(&lock->wait_lock);
+			__raw_spin_unlock(&task->pi_lock);
 			trace_unlock_irq(&trace_lock, ti);
 			break;
 		}
@@ -1741,6 +1848,7 @@
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
+	__raw_spin_lock(&task->pi_lock);
 	__raw_spin_lock(&lock->wait_lock);
 
 	old_owner = lock_owner(lock);
@@ -1749,13 +1857,18 @@
 	if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 		/* granted */
 		TRACE_WARN_ON_LOCKED(!plist_empty(&lock->wait_list) && !old_owner);
-		__raw_spin_lock(&pi_lock);
-		set_new_owner(lock, old_owner, ti __EIP__);
-		__raw_spin_unlock(&pi_lock);
+		if (old_owner) {
+			__raw_spin_lock(&old_owner->task->pi_lock);
+			set_new_owner(lock, old_owner, ti __EIP__);
+			__raw_spin_unlock(&old_owner->task->pi_lock);
+		} else
+			set_new_owner(lock, old_owner, ti __EIP__);
+		__raw_spin_unlock(&lock->wait_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		ret = 1;
 	}
-
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 	return ret;
@@ -1817,6 +1930,7 @@
 	 * to the previous priority (or to the next highest prio
 	 * waiter's priority):
 	 */
+	__raw_spin_lock(&old_owner->pi_lock);
 	prio = old_owner->normal_prio;
 	if (unlikely(!plist_empty(&old_owner->pi_waiters))) {
 		w = plist_first_entry(&old_owner->pi_waiters, struct rt_mutex_waiter, pi_list);
@@ -1825,6 +1939,7 @@
 	}
 	if (unlikely(prio != old_owner->prio))
 		pi_setprio(lock, old_owner, prio);
+	__raw_spin_unlock(&old_owner->pi_lock);
 #ifdef CAPTURE_LOCK
 	if (lock != &kernel_sem.lock) {
 		new_owner->rt_flags |= RT_PENDOWNER;
@@ -1851,6 +1966,7 @@
 	 * to the previous priority (or to the next highest prio
 	 * waiter's priority):
 	 */
+	__raw_spin_lock(&old_owner->pi_lock);
 	prio = old_owner->normal_prio;
 	if (unlikely(!plist_empty(&old_owner->pi_waiters))) {
 		w = plist_first_entry(&old_owner->pi_waiters, struct rt_mutex_waiter, pi_list);
@@ -1859,6 +1975,7 @@
 	}
 	if (unlikely(prio != old_owner->prio))
 		pi_setprio(lock, old_owner, prio);
+	__raw_spin_unlock(&old_owner->pi_lock);
 #ifdef CAPTURE_LOCK
 	if (lock != &kernel_sem.lock) {
 		new_owner->rt_flags |= RT_PENDOWNER;

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Thu, 2005-08-25 at 10:47 -0400, Steven Rostedt wrote:
> OK, here it is. Against 2.6.13-rc7-rt1. 

> @@ -1134,17 +1206,35 @@
>  		return 0;
>  
>  	trace_lock_irqsave(&trace_lock, flags, ti);
> +	/*
> +	 * We are no longer blocked on the lock, so we are considered a
> +	 * owner. So we must grab the lock->wait_lock first.
> +	 */
>  	__raw_spin_lock(&lock->wait_lock);
> +	__raw_spin_lock(&task->pi_lock);
>  
>  	if (!(task->rt_flags & RT_PENDOWNER)) {
> -		/* someone else stole it */
> +		/*
> +		 * Someone else stole it.
> +		 * We are grabbing a lock now, so we must release the
> +		 * locks again and retake them in the opposite order.
> +		 */
> +		__raw_spin_unlock(&task->pi_lock);
> +		__raw_spin_unlock(&lock->wait_lock);
> +
> +		__raw_spin_lock(&task->pi_lock);
> +		__raw_spin_lock(&lock->wait_lock);
> +

The above code is added to capture_lock, and I don't like it.  I was
thinking that since the locking order was to be:

P1 blocked_on L1 owned_by P2 blocked_on ... Ln-1 owned by Pn

So you always need to grab the process lock before grabbing the mutex
lock that it is blocked on, and always grab the mutex lock before
grabbing the process lock that owns it.

Here it is a little ambiguous.  The process use to own the lock, but
someone stole it.  When grabbing a lock, I always grab the process lock
first before grabbing the lock's lock, but this isn't necessary.  So if
you already have the two locks (mutex and process) as is the case above,
you don't need to unlock them and regrab them (although this doesn't
hurt, except for performance), because any race would have been with the
grabbing of these two locks in the first place.

Now, here's why it's safe.  The process that use to own the mutex and
had it stolen now is out of the chain.  The pi_lock and the wait_lock
here are not in any order.  So no one who is grabbing the process'
pi_lock should have owned the wait_lock and vice versa.

So here's an updated patch without this lock switching:

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_goliath/kernel/rt.c
===================================================================
--- linux_realtime_goliath/kernel/rt.c	(revision 304)
+++ linux_realtime_goliath/kernel/rt.c	(working copy)
@@ -22,6 +22,9 @@
  *
  *  Copyright (C) 2005, Kihon Technologies Inc., Steven Rostedt
  *
+ *   (also by Steven Rostedt)
+ *    - Converted single pi_lock to individual task locks.
+ *
  */
 #include <linux/config.h>
 #include <linux/sched.h>
@@ -36,6 +39,26 @@
 #define CAPTURE_LOCK
 
 /*
+ * Lock order:
+ *
+ *  To keep from having a single lock for PI, each task and lock
+ *  has their own locking. The order is as follows:
+ *
+ * blocked task->pi_lock -> lock->wait_lock -> owner task->pi_lock.
+ * 
+ * This is safe since a owner task should never block on a lock that
+ * is owned by a blocking task.  Otherwise you would have a deadlock 
+ * in the normal system.
+ * The same goes for the locks. A lock held by one task, should not be
+ * taken by task that holds a lock that is blocking this lock's owner.
+ *
+ * A task that is about to grab a lock is first considered to be a 
+ * blocking task, even if the task successfully acquires the lock.
+ * This is because the taking of the locks happen before the
+ * task becomes the owner.
+ */
+
+/*
  * These flags are used for allowing of stealing of ownerships.
  */
 #define RT_PENDOWNER	1	/* pending owner on a lock */
@@ -83,14 +106,6 @@
  */
 //#define ALL_TASKS_PI
 
-/*
- * We need a global lock for priority inheritance handling.
- * This is only for the slow path, but still, we might want
- * to optimize it later to be more scalable.
- */
-static __cacheline_aligned_in_smp raw_spinlock_t pi_lock =
-						RAW_SPIN_LOCK_UNLOCKED;
-
 #ifdef CONFIG_RT_DEADLOCK_DETECT
 # define __EIP_DECL__ , unsigned long eip
 # define __EIP__ , eip
@@ -174,6 +189,8 @@
 	if (trace_on) {				\
 		trace_on = 0;			\
 		console_verbose();		\
+		if (spin_is_locked(&current->pi_lock)) \
+			__raw_spin_unlock(&current->pi_lock); \
 		spin_unlock(&trace_lock);	\
 	}					\
 } while (0)
@@ -228,7 +245,6 @@
  */
 void zap_rt_locks(void)
 {
-	spin_lock_init(&pi_lock);
 #ifdef CONFIG_RT_DEADLOCK_DETECT
 	spin_lock_init(&trace_lock);
 #endif
@@ -589,11 +605,11 @@
 			printk("exiting task is not even the owner??\n");
 		goto restart;
 	}
-	__raw_spin_lock(&pi_lock);
+	__raw_spin_lock(&task->pi_lock);
 	plist_for_each(curr1, &task->pi_waiters) {
 		w = plist_entry(curr1, struct rt_mutex_waiter, pi_list);
 		TRACE_OFF();
-		__raw_spin_unlock(&pi_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 		printk("hm, PI interest held at exit time? Task:\n");
@@ -601,7 +617,7 @@
 		printk_waiter(w);
 		return;
 	}
-	__raw_spin_unlock(&pi_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 }
 
@@ -674,6 +690,7 @@
 	struct rt_mutex_waiter *w;
 	struct plist *curr1;
 
+	__raw_spin_lock(&old_owner->task->pi_lock);
 	plist_for_each(curr1, &old_owner->task->pi_waiters) {
 		w = plist_entry(curr1, struct rt_mutex_waiter, pi_list);
 		if (w->lock == lock) {
@@ -682,9 +699,10 @@
 			printk_waiter(w);
 			printk("\n");
 			TRACE_WARN_ON(1);
-			return;
+			break;
 		}
 	}
+	__raw_spin_unlock(&old_owner->task->pi_lock);
 }
 
 #else
@@ -732,13 +750,24 @@
 
 int pi_walk, pi_null, pi_prio, pi_initialized;
 
-static void pi_setprio(struct rt_mutex *lock, struct task_struct *p, int prio)
+/*
+ * The lock->wait_lock and p->pi_lock must be held.
+ */
+static void pi_setprio(struct rt_mutex *lock, struct task_struct *task, int prio)
 {
+	struct rt_mutex *l = lock;
+	struct task_struct *p = task;
+	/*
+	 * We don't want to release the parameters locks.
+	 */
+
 	if (unlikely(!p->pid)) {
 		pi_null++;
 		return;
 	}
 
+	TRACE_BUG_ON_LOCKED(!spin_is_locked(&lock->wait_lock));
+	TRACE_BUG_ON_LOCKED(!spin_is_locked(&p->pi_lock));
 #ifdef CONFIG_RT_DEADLOCK_DETECT
 	pi_prio++;
 	if (p->policy != SCHED_NORMAL && prio > normal_prio(p)) {
@@ -759,7 +788,6 @@
 	 * If the task is blocked on some other task then boost that
 	 * other task (or tasks) too:
 	 */
-	__raw_spin_lock(&p->pi_lock);
 	for (;;) {
 		struct rt_mutex_waiter *w = p->blocked_on;
 #ifdef CONFIG_RT_DEADLOCK_DETECT
@@ -774,16 +802,27 @@
 		 * it RT, then register the task in the PI list and
 		 * requeue it to the wait list:
 		 */
-		lock = w->lock;
+
+		/*
+		 * Don't unlock the original lock->wait_lock
+		 */
+		if (l != lock)
+			__raw_spin_unlock(&l->wait_lock);
+		l = w->lock;
 		TRACE_BUG_ON_LOCKED(!lock);
-		TRACE_BUG_ON_LOCKED(!lock_owner(lock));
+
+		if (l != lock)
+			__raw_spin_lock(&l->wait_lock);
+
+		TRACE_BUG_ON_LOCKED(!lock_owner(l));
 		if (rt_task(p) && plist_empty(&w->pi_list)) {
+			TRACE_BUG_ON_LOCKED(was_rt);
 			plist_init(&w->pi_list, prio);
-			plist_add(&w->pi_list, &lock_owner(lock)->task->pi_waiters);
+			plist_add(&w->pi_list, &lock_owner(l)->task->pi_waiters);
 
-			plist_del(&w->list, &lock->wait_list);
+			plist_del(&w->list, &l->wait_list);
 			plist_init(&w->list, prio);
-			plist_add(&w->list, &lock->wait_list);
+			plist_add(&w->list, &l->wait_list);
 		}
 		/*
 		 * If the task is blocked on a lock, and we just restored
@@ -795,16 +834,18 @@
 		 */
 		if (!rt_task(p) && !plist_empty(&w->pi_list)) {
 			TRACE_BUG_ON_LOCKED(!was_rt);
-			plist_del(&w->pi_list, &lock_owner(lock)->task->pi_waiters);
-			plist_del(&w->list, &lock->wait_list);
+			plist_del(&w->pi_list, &lock_owner(l)->task->pi_waiters);
+			plist_del(&w->list, &l->wait_list);
 			plist_init(&w->list, prio);
-			plist_add(&w->list, &lock->wait_list);
+			plist_add(&w->list, &l->wait_list);
 		}
 
 		pi_walk++;
+		
+		if (p != task)
+			__raw_spin_unlock(&p->pi_lock);
 
-		__raw_spin_unlock(&p->pi_lock);
-		p = lock_owner(lock)->task;
+		p = lock_owner(l)->task;
 		TRACE_BUG_ON_LOCKED(!p);
 		__raw_spin_lock(&p->pi_lock);
 		/*
@@ -815,7 +856,10 @@
 		if (p->prio <= prio)
 			break;
 	}
-	__raw_spin_unlock(&p->pi_lock);
+	if (l != lock)
+		__raw_spin_unlock(&l->wait_lock);
+	if (p != task)
+		__raw_spin_unlock(&p->pi_lock);
 }
 
 /*
@@ -831,7 +875,9 @@
 	unsigned long flags;
 	int oldprio;
 
-	spin_lock_irqsave(&pi_lock, flags);
+	spin_lock_irqsave(&p->pi_lock,flags);
+	if (p->blocked_on)
+		spin_lock(&p->blocked_on->lock->wait_lock);
 
 	oldprio = p->normal_prio;
 	if (oldprio == prio)
@@ -858,10 +904,17 @@
 	else
 		mutex_setprio(p, prio);
  out:
-	spin_unlock_irqrestore(&pi_lock, flags);
+	if (p->blocked_on)
+		spin_unlock(&p->blocked_on->lock->wait_lock);
 
+	spin_unlock_irqrestore(&p->pi_lock, flags);
+
 }
 
+/*
+ * This is called with both the waiter->task->pi_lock and 
+ * lock->wait_lock held.
+ */
 static void
 task_blocks_on_lock(struct rt_mutex_waiter *waiter, struct thread_info *ti,
 		    struct rt_mutex *lock __EIP_DECL__)
@@ -872,7 +925,6 @@
 	/* mark the current thread as blocked on the lock */
 	waiter->eip = eip;
 #endif
-	__raw_spin_lock(&task->pi_lock);
 	task->blocked_on = waiter;
 	waiter->lock = lock;
 	waiter->ti = ti;
@@ -884,31 +936,24 @@
 	if (!rt_task(task)) {
 		plist_add(&waiter->list, &lock->wait_list);
 		set_lock_owner_pending(lock);
-		__raw_spin_unlock(&task->pi_lock);
 		return;
 	}
 #endif
-	__raw_spin_unlock(&task->pi_lock);
-	__raw_spin_lock(&pi_lock);
+	plist_add(&waiter->pi_list, &lock_owner(lock)->task->pi_waiters);
 	/*
-	 * We could have been added to the pi list from another task
-	 * doing a pi_setprio.
+	 * Add RT tasks to the head:
 	 */
-	if (likely(plist_empty(&waiter->pi_list))) {
-		plist_add(&waiter->pi_list, &lock_owner(lock)->task->pi_waiters);
-		/*
-		 * Add RT tasks to the head:
-		 */
-		plist_add(&waiter->list, &lock->wait_list);
-	}
+	plist_add(&waiter->list, &lock->wait_list);
 	set_lock_owner_pending(lock);
 	/*
 	 * If the waiter has higher priority than the owner
 	 * then temporarily boost the owner:
 	 */
-	if (task->prio < lock_owner(lock)->task->prio)
+	if (task->prio < lock_owner(lock)->task->prio) {
+		__raw_spin_lock(&lock_owner(lock)->task->pi_lock);
 		pi_setprio(lock, lock_owner(lock)->task, task->prio);
-	__raw_spin_unlock(&pi_lock);
+		__raw_spin_unlock(&lock_owner(lock)->task->pi_lock);
+	}
 }
 
 /*
@@ -942,6 +987,10 @@
 }
 EXPORT_SYMBOL(__init_rwsem);
 
+/*
+ * This must be called with both the old_owner and new_owner pi_locks held.
+ * As well as the lock->wait_lock.
+ */
 static inline
 void set_new_owner(struct rt_mutex *lock, struct thread_info *old_owner,
 			struct thread_info *new_owner __EIP_DECL__)
@@ -975,6 +1024,8 @@
 /*
  * handle the lock release when processes blocked on it that can now run
  * - the spinlock must be held by the caller
+ *
+ * The lock->wait_lock must be held, and the lock's owner->pi_lock must not.
  */
 static struct thread_info *
 pick_new_owner(struct rt_mutex *lock, struct thread_info *old_owner,
@@ -990,14 +1041,34 @@
 	 */
 	waiter = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, list);
 
+ try_again:
 	trace_special_pid(waiter->ti->task->pid, waiter->ti->task->prio, 0);
 
 #ifdef ALL_TASKS_PI
 	check_pi_list_present(lock, waiter, old_owner);
 #endif
 	new_owner = waiter->ti;
+	/*
+	 * The new owner is still blocked on this lock, so we
+	 * must release the lock->wait_lock before grabing 
+	 * the new_owner lock.
+	 */
+	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_lock(&new_owner->task->pi_lock);
+	__raw_spin_lock(&lock->wait_lock);
+	/*
+	 * In this split second of releasing the lock, a high priority 
+	 * process could have come along and blocked as well.
+	 */
+	waiter = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, list);
+	if (unlikely(waiter->ti != new_owner)) {
+		__raw_spin_unlock(&new_owner->task->pi_lock);
+		goto try_again;
+	}
 	plist_del_init(&waiter->list, &lock->wait_list);
 
+	__raw_spin_lock(&old_owner->task->pi_lock);
+
 	plist_del(&waiter->pi_list, &old_owner->task->pi_waiters);
 	plist_init(&waiter->pi_list, waiter->ti->task->prio);
 
@@ -1008,6 +1079,9 @@
 	new_owner->task->blocked_on = NULL;
 	TRACE_WARN_ON(save_state != lock->save_state);
 
+	__raw_spin_unlock(&old_owner->task->pi_lock);
+	__raw_spin_unlock(&new_owner->task->pi_lock);
+
 	return new_owner;
 }
 
@@ -1060,9 +1134,7 @@
 	 * is still sleeping and hasn't woken up to get the lock.
 	 */
 
-	/* Test the simple case first, is it already running? */
-	if (!TASK_PENDING(owner))
-		return 0;
+	TRACE_BUG_ON_LOCKED(!owner);
 
 	/* The owner is pending on a lock, but is it this lock? */
 	if (owner->pending_owner != lock)
@@ -1134,17 +1206,27 @@
 		return 0;
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
+	/*
+	 * We are no longer blocked on the lock, so we are considered a
+	 * owner. So we must grab the lock->wait_lock first.
+	 */
 	__raw_spin_lock(&lock->wait_lock);
+	__raw_spin_lock(&task->pi_lock);
 
 	if (!(task->rt_flags & RT_PENDOWNER)) {
-		/* someone else stole it */
+		/*
+		 * Someone else stole it.
+		 */
 		old_owner = lock_owner(lock);
 		TRACE_BUG_ON_LOCKED(old_owner == ti);
 		if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 			/* we got it back! */
-			__raw_spin_lock(&pi_lock);
-			set_new_owner(lock, old_owner, ti __W_EIP__(waiter));
-			__raw_spin_unlock(&pi_lock);
+			if (old_owner) {
+				__raw_spin_lock(&old_owner->task->pi_lock);
+				set_new_owner(lock, old_owner, ti __W_EIP__(waiter));
+				__raw_spin_unlock(&old_owner->task->pi_lock);
+			} else
+				set_new_owner(lock, old_owner, ti __W_EIP__(waiter));
 			ret = 0;
 		} else {
 			/* Add ourselves back to the list */
@@ -1159,6 +1241,7 @@
 	}
 
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 	return ret;
@@ -1196,6 +1279,7 @@
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
+	__raw_spin_lock(&task->pi_lock);
 	__raw_spin_lock(&lock->wait_lock);
 	INIT_WAITER(&waiter);
 
@@ -1205,10 +1289,14 @@
 	if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 		/* granted */
 		TRACE_WARN_ON_LOCKED(!plist_empty(&lock->wait_list) && !old_owner);
-		__raw_spin_lock(&pi_lock);
-		set_new_owner(lock, old_owner, ti __EIP__);
-		__raw_spin_unlock(&pi_lock);
+		if (old_owner) {
+			__raw_spin_lock(&old_owner->task->pi_lock);
+			set_new_owner(lock, old_owner, ti __EIP__);
+			__raw_spin_unlock(&old_owner->task->pi_lock);
+		} else
+			set_new_owner(lock, old_owner, ti __EIP__);
 		__raw_spin_unlock(&lock->wait_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 		FREE_WAITER(&waiter);
@@ -1223,6 +1311,7 @@
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
 	/* we don't need to touch the lock struct anymore */
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 	might_sleep();
@@ -1273,6 +1362,7 @@
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
+	__raw_spin_lock(&task->pi_lock);
 	__raw_spin_lock(&lock->wait_lock);
 	INIT_WAITER(&waiter);
 
@@ -1282,10 +1372,14 @@
 	if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 		/* granted */
 		TRACE_WARN_ON_LOCKED(!plist_empty(&lock->wait_list) && !old_owner);
-		__raw_spin_lock(&pi_lock);
-		set_new_owner(lock, old_owner, ti __EIP__);
-		__raw_spin_unlock(&pi_lock);
+		if (old_owner) {
+			__raw_spin_lock(&old_owner->task->pi_lock);
+			set_new_owner(lock, old_owner, ti __EIP__);
+			__raw_spin_unlock(&old_owner->task->pi_lock);
+		} else
+			set_new_owner(lock, old_owner, ti __EIP__);
 		__raw_spin_unlock(&lock->wait_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 		FREE_WAITER(&waiter);
@@ -1306,6 +1400,7 @@
 
 	/* we don't need to touch the lock struct anymore */
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock(&trace_lock, ti);
 
 	/*
@@ -1396,7 +1491,6 @@
 	TRACE_WARN_ON_LOCKED(list_empty(&lock->held_list));
 	list_del_init(&lock->held_list);
 #endif
-	__raw_spin_lock(&pi_lock);
 
 #ifdef ALL_TASKS_PI
 	if (plist_empty(&lock->wait_list))
@@ -1409,7 +1503,6 @@
 			__up_mutex_waiter_nosavestate(lock __EIP__);
 	} else
 		lock->owner = NULL;
-	__raw_spin_unlock(&pi_lock);
 	__raw_spin_unlock(&lock->wait_lock);
 #ifdef CONFIG_DEBUG_PREEMPT
 	if (current->lock_count < 0 || current->lock_count >= MAX_LOCK_STACK) {
@@ -1640,6 +1733,7 @@
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
+	__raw_spin_lock(&task->pi_lock);
 	__raw_spin_lock(&lock->wait_lock);
 	INIT_WAITER(&waiter);
 
@@ -1649,10 +1743,14 @@
 	if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 		/* granted */
 		TRACE_WARN_ON_LOCKED(!plist_empty(&lock->wait_list) && !old_owner);
-		__raw_spin_lock(&pi_lock);
-		set_new_owner(lock, old_owner, ti __EIP__);
-		__raw_spin_unlock(&pi_lock);
+		if (old_owner) {
+			__raw_spin_lock(&old_owner->task->pi_lock);
+			set_new_owner(lock, old_owner, ti __EIP__);
+			__raw_spin_unlock(&old_owner->task->pi_lock);
+		} else
+			set_new_owner(lock, old_owner, ti __EIP__);
 		__raw_spin_unlock(&lock->wait_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 		FREE_WAITER(&waiter);
@@ -1667,6 +1765,7 @@
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
 	/* we don't need to touch the lock struct anymore */
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 	might_sleep();
@@ -1684,6 +1783,7 @@
 			 * didnt get the lock - else return success:
 			 */
 			trace_lock_irq(&trace_lock, ti);
+			__raw_spin_lock(&task->pi_lock);
 			__raw_spin_lock(&lock->wait_lock);
 			if (waiter.ti) {
 				plist_del_init(&waiter.list, &lock->wait_list);
@@ -1692,17 +1792,16 @@
 				 * (No big problem if our PI effect lingers
 				 *  a bit - owner will restore prio.)
 				 */
-				__raw_spin_lock(&pi_lock);
 				TRACE_WARN_ON_LOCKED(waiter.ti != ti);
 				TRACE_WARN_ON_LOCKED(current->blocked_on != &waiter);
 				plist_del(&waiter.pi_list, &task->pi_waiters);
 				plist_init(&waiter.pi_list, task->prio);
 				waiter.ti = NULL;
 				current->blocked_on = NULL;
-				__raw_spin_unlock(&pi_lock);
 				ret = -EINTR;
 			}
 			__raw_spin_unlock(&lock->wait_lock);
+			__raw_spin_unlock(&task->pi_lock);
 			trace_unlock_irq(&trace_lock, ti);
 			break;
 		}
@@ -1741,6 +1840,7 @@
 
 	trace_lock_irqsave(&trace_lock, flags, ti);
 	TRACE_BUG_ON_LOCKED(!raw_irqs_disabled());
+	__raw_spin_lock(&task->pi_lock);
 	__raw_spin_lock(&lock->wait_lock);
 
 	old_owner = lock_owner(lock);
@@ -1749,13 +1849,18 @@
 	if (likely(!old_owner) || __grab_lock(lock, task, old_owner->task)) {
 		/* granted */
 		TRACE_WARN_ON_LOCKED(!plist_empty(&lock->wait_list) && !old_owner);
-		__raw_spin_lock(&pi_lock);
-		set_new_owner(lock, old_owner, ti __EIP__);
-		__raw_spin_unlock(&pi_lock);
+		if (old_owner) {
+			__raw_spin_lock(&old_owner->task->pi_lock);
+			set_new_owner(lock, old_owner, ti __EIP__);
+			__raw_spin_unlock(&old_owner->task->pi_lock);
+		} else
+			set_new_owner(lock, old_owner, ti __EIP__);
+		__raw_spin_unlock(&lock->wait_lock);
+		__raw_spin_unlock(&task->pi_lock);
 		ret = 1;
 	}
-
 	__raw_spin_unlock(&lock->wait_lock);
+	__raw_spin_unlock(&task->pi_lock);
 	trace_unlock_irqrestore(&trace_lock, flags, ti);
 
 	return ret;
@@ -1817,6 +1922,7 @@
 	 * to the previous priority (or to the next highest prio
 	 * waiter's priority):
 	 */
+	__raw_spin_lock(&old_owner->pi_lock);
 	prio = old_owner->normal_prio;
 	if (unlikely(!plist_empty(&old_owner->pi_waiters))) {
 		w = plist_first_entry(&old_owner->pi_waiters, struct rt_mutex_waiter, pi_list);
@@ -1825,6 +1931,7 @@
 	}
 	if (unlikely(prio != old_owner->prio))
 		pi_setprio(lock, old_owner, prio);
+	__raw_spin_unlock(&old_owner->pi_lock);
 #ifdef CAPTURE_LOCK
 	if (lock != &kernel_sem.lock) {
 		new_owner->rt_flags |= RT_PENDOWNER;
@@ -1851,6 +1958,7 @@
 	 * to the previous priority (or to the next highest prio
 	 * waiter's priority):
 	 */
+	__raw_spin_lock(&old_owner->pi_lock);
 	prio = old_owner->normal_prio;
 	if (unlikely(!plist_empty(&old_owner->pi_waiters))) {
 		w = plist_first_entry(&old_owner->pi_waiters, struct rt_mutex_waiter, pi_list);
@@ -1859,6 +1967,7 @@
 	}
 	if (unlikely(prio != old_owner->prio))
 		pi_setprio(lock, old_owner, prio);
+	__raw_spin_unlock(&old_owner->pi_lock);
 #ifdef CAPTURE_LOCK
 	if (lock != &kernel_sem.lock) {
 		new_owner->rt_flags |= RT_PENDOWNER;

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Thu, 2005-08-25 at 08:35 +0200, Ingo Molnar wrote:
> * Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > Well, after turning off hrtimers, I keep getting one bug. A possible 
> > soft lockup with the ext3 code. But this didn't seem to be caused by 
> > the changes I made. So just to be sure, I ran my test on the vanilla 
> > 2.6.13-rc6-rt11 and it gave the same bug too.  So, it looks like my 
> > changes are now at par with what is out there with the rt11 release. 
> > They both give the same bug! ;-)
> 
> does the system truly lock up, or is this some transitional condition?  
> In any case, i agree that this should be debugged independently of the 
> pi_lock patch.

Hmm, I forgot that you took out the bit_spin_lock fixes.  I think this
may be caused by them.  I haven't look further into it yet. 

Oh, and I'm sending you this on your latest patch with my pi_lock patch
applied. (no debugging turned on either and this is an SMP machine).

-- Steve

Re: [RFC] RT-patch update to remove the global pi_lock

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> Here it is a little ambiguous.  The process use to own the lock, but 
> someone stole it.  When grabbing a lock, I always grab the process 
> lock first before grabbing the lock's lock, but this isn't necessary.  
> So if you already have the two locks (mutex and process) as is the 
> case above, you don't need to unlock them and regrab them (although 
> this doesn't hurt, except for performance), because any race would 
> have been with the grabbing of these two locks in the first place.
> 
> Now, here's why it's safe.  The process that use to own the mutex and 
> had it stolen now is out of the chain.  The pi_lock and the wait_lock 
> here are not in any order.  So no one who is grabbing the process' 
> pi_lock should have owned the wait_lock and vice versa.
> 
> So here's an updated patch without this lock switching:

your patch works great here, on 3 separate systems: a 1-way, a 2/4-way 
and an 8-way.

the 1-way system performed so well running the SMP kernel that i first 
thought i booted the UP kernel by accident :-)

on the 8-way box, "hackbench 10" got _3.7_ times faster (!!!).

i have booted the 8-way box without your patch once more because i didnt 
believe the results initially and thought they were some benchmarking 
fluke. But no, it wasnt a fluke. The kernel profiles are nicely flat 
now.

i've released 2.6.13-rc7-rt2 with your patch included. This is certainly 
a major milestone for PREEMPT_RT, it is now a first-class scalability 
citizen on SMP too. Great work Steven!

	Ingo

Re: [RFC] RT-patch update to remove the global pi_lock

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> > does the system truly lock up, or is this some transitional condition?  
> > In any case, i agree that this should be debugged independently of the 
> > pi_lock patch.
> 
> Hmm, I forgot that you took out the bit_spin_lock fixes.  I think this 
> may be caused by them.  I haven't look further into it yet.

yeah, i took them out because they clashed with upstream changes. Note 
that i meanwhile also introduced a per-bh lock, which might make it 
easier to fix the deadlock:

 --- linux.orig/fs/buffer.c
 +++ linux/fs/buffer.c
 @@ -537,8 +537,7 @@ static void end_buffer_async_read(struct
          * decide that the page is now completely done.
          */
         first = page_buffers(page);
 -       local_irq_save(flags);
 -       bit_spin_lock(BH_Uptodate_Lock, &first->b_state);
 +       spin_lock_irqsave(&first->b_uptodate_lock, flags);
         clear_buffer_async_read(bh);
         unlock_buffer(bh);
         tmp = bh;

could jbd reuse this lock - or would it need another lock?

	Ingo

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Thu, 2005-08-25 at 21:34 +0200, Ingo Molnar wrote:
> * Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > > does the system truly lock up, or is this some transitional condition?  
> > > In any case, i agree that this should be debugged independently of the 
> > > pi_lock patch.
> > 
> > Hmm, I forgot that you took out the bit_spin_lock fixes.  I think this 
> > may be caused by them.  I haven't look further into it yet.
> 
> yeah, i took them out because they clashed with upstream changes. Note 
> that i meanwhile also introduced a per-bh lock, which might make it 
> easier to fix the deadlock:
> 
>  --- linux.orig/fs/buffer.c
>  +++ linux/fs/buffer.c
>  @@ -537,8 +537,7 @@ static void end_buffer_async_read(struct
>           * decide that the page is now completely done.
>           */
>          first = page_buffers(page);
>  -       local_irq_save(flags);
>  -       bit_spin_lock(BH_Uptodate_Lock, &first->b_state);
>  +       spin_lock_irqsave(&first->b_uptodate_lock, flags);
>          clear_buffer_async_read(bh);
>          unlock_buffer(bh);
>          tmp = bh;
> 
> could jbd reuse this lock - or would it need another lock?

I think it can.  I'm looking into right now, but first I'm updating my
logdev to the latest release.  I stripped it all out after submitting
that pi_lock patch and now I have to put it back in!   I didn't save the
updates that I added earlier, so I'm reworking things now.  The logging
definitely helps me, since that was a major factor in getting that
pi_lock patch done so quick.

-- Steve

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Thu, 2005-08-25 at 19:47 +0200, Ingo Molnar wrote:

> your patch works great here, on 3 separate systems: a 1-way, a 2/4-way 
> and an 8-way.
> 
> the 1-way system performed so well running the SMP kernel that i first 
> thought i booted the UP kernel by accident :-)
> 
> on the 8-way box, "hackbench 10" got _3.7_ times faster (!!!).
> 
> i have booted the 8-way box without your patch once more because i didnt 
> believe the results initially and thought they were some benchmarking 
> fluke. But no, it wasnt a fluke. The kernel profiles are nicely flat 
> now.
> 
> i've released 2.6.13-rc7-rt2 with your patch included. This is certainly 
> a major milestone for PREEMPT_RT, it is now a first-class scalability 
> citizen on SMP too. Great work Steven!
> 
> 	Ingo

A word of caution (aka. disclaimer). This is still new.  I still expect
there are some cases in the code that was missed and can cause a dead
lock or other bad side effect.  Hopefully, we can iron these all out.
Also, I noticed that since the task takes it's own pi_lock for most of
the code, if something locks up and a NMI goes off, the down_trylock in
printk will also lock when it tries to take it's own pi_lock.

I stated earlier, that I converted printk to use early_printk (serial)
on a oops_in_progress, so I wouldn't need to worry about the unlocking
circus that needs to be done.  So, I'm sure if something goes wrong now,
you won't see anything, even with an NMI.

If someone else would like to fix that, I'm sure Ingo would be willing
to accept patches :-)

-- Steve

Re: [RFC] RT-patch update to remove the global pi_lock

[Daniel Walker]

On Thu, 2005-08-25 at 16:09 -0400, Steven Rostedt wrote:

> A word of caution (aka. disclaimer). This is still new.  I still expect
> there are some cases in the code that was missed and can cause a dead
> lock or other bad side effect.  Hopefully, we can iron these all out.
> Also, I noticed that since the task takes it's own pi_lock for most of
> the code, if something locks up and a NMI goes off, the down_trylock in
> printk will also lock when it tries to take it's own pi_lock.

maybe it's time for ALL_TASKS_PI ?


Daniel

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Thu, 2005-08-25 at 16:09 -0400, Steven Rostedt wrote:

> A word of caution (aka. disclaimer). This is still new.  I still expect
> there are some cases in the code that was missed and can cause a dead
> lock or other bad side effect.  Hopefully, we can iron these all out.
> Also, I noticed that since the task takes it's own pi_lock for most of
> the code, if something locks up and a NMI goes off, the down_trylock in
> printk will also lock when it tries to take it's own pi_lock.

OK, found my first bug :-)

Just so everyone knows.  In rt.c, all pi_waiter access (reading or
writing) must be protected by the task's pi_lock, and all access to the
lock's wait_list must be protected by the lock's wait_lock.  The magic
is in the locking order :-).

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_goliath/kernel/rt.c
===================================================================
--- linux_realtime_goliath/kernel/rt.c	(revision 306)
+++ linux_realtime_goliath/kernel/rt.c	(working copy)
@@ -671,6 +671,7 @@
 	struct rt_mutex_waiter *w;
 	struct plist *curr1;
 
+	__raw_spin_lock(old_owner->task->pi_lock);
 	TRACE_WARN_ON_LOCKED(plist_empty(&waiter->pi_list));
 	TRACE_WARN_ON_LOCKED(lock_owner(lock));
 
@@ -681,6 +682,7 @@
 	}
 	TRACE_WARN_ON_LOCKED(1);
 ok:
+	__raw_spin_unlock(old_owner->task->pi_lock);
 	return;
 }
 
@@ -734,6 +736,8 @@
 	if (old_owner == new_owner)
 		return;
 
+	TRACE_BUG_ON_LOCKED(!spin_is_locked(&old_owner->task->pi_lock));
+	TRACE_BUG_ON_LOCKED(!spin_is_locked(&new_owner->task->pi_lock));
 	plist_for_each_safe(curr1, next1, &old_owner->task->pi_waiters) {
 		w = plist_entry(curr1, struct rt_mutex_waiter, pi_list);
 		if (w->lock == lock) {
@@ -932,6 +936,8 @@
 	/*
 	 * Add SCHED_NORMAL tasks to the end of the waitqueue (FIFO):
 	 */
+	TRACE_BUG_ON_LOCKED(!spin_is_locked(&task->pi_lock));
+	TRACE_BUG_ON_LOCKED(!spin_is_locked(&lock->wait_lock));
 #ifndef ALL_TASKS_PI
 	if (!rt_task(task)) {
 		plist_add(&waiter->list, &lock->wait_list);
@@ -939,6 +945,7 @@
 		return;
 	}
 #endif
+	__raw_spin_lock(&lock_owner(lock)->task->pi_lock);
 	plist_add(&waiter->pi_list, &lock_owner(lock)->task->pi_waiters);
 	/*
 	 * Add RT tasks to the head:
@@ -949,11 +956,9 @@
 	 * If the waiter has higher priority than the owner
 	 * then temporarily boost the owner:
 	 */
-	if (task->prio < lock_owner(lock)->task->prio) {
-		__raw_spin_lock(&lock_owner(lock)->task->pi_lock);
+	if (task->prio < lock_owner(lock)->task->prio)
 		pi_setprio(lock, lock_owner(lock)->task, task->prio);
-		__raw_spin_unlock(&lock_owner(lock)->task->pi_lock);
-	}
+	__raw_spin_unlock(&lock_owner(lock)->task->pi_lock);
 }
 
 /*

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

Ingo,

The following code segment from pick_new_owner:


	waiter = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, list);

 try_again:
	trace_special_pid(waiter->ti->task->pid, waiter->ti->task->prio, 0);

#ifdef ALL_TASKS_PI
	check_pi_list_present(lock, waiter, old_owner);
#endif
	new_owner = waiter->ti;
	/*
	 * The new owner is still blocked on this lock, so we
	 * must release the lock->wait_lock before grabing 
	 * the new_owner lock.
	 */
	__raw_spin_unlock(&lock->wait_lock);
	__raw_spin_lock(&new_owner->task->pi_lock);
	__raw_spin_lock(&lock->wait_lock);
	/*
	 * In this split second of releasing the lock, a high priority 
	 * process could have come along and blocked as well.
	 */
	waiter = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, list);
	if (unlikely(waiter->ti != new_owner)) {
		__raw_spin_unlock(&new_owner->task->pi_lock);
		goto try_again;
	}


Is basically a waste on UP. Should the following patch be applied
instead?


--- linux_realtime_ernie/kernel/rt.c.orig	2005-08-26 09:46:34.000000000 -0400
+++ linux_realtime_ernie/kernel/rt.c	2005-08-26 09:48:17.000000000 -0400
@@ -1046,7 +1046,9 @@ pick_new_owner(struct rt_mutex *lock, st
 	 */
 	waiter = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, list);
 
+#ifdef CONFIG_SMP
  try_again:
+#endif
 	trace_special_pid(waiter->ti->task->pid, waiter->ti->task->prio, 0);
 
 #ifdef ALL_TASKS_PI
@@ -1065,11 +1067,13 @@ pick_new_owner(struct rt_mutex *lock, st
 	 * In this split second of releasing the lock, a high priority 
 	 * process could have come along and blocked as well.
 	 */
+#ifdef CONFIG_SMP
 	waiter = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, list);
 	if (unlikely(waiter->ti != new_owner)) {
 		__raw_spin_unlock(&new_owner->task->pi_lock);
 		goto try_again;
 	}
+#endif
 	plist_del_init(&waiter->list, &lock->wait_list);
 
 	__raw_spin_lock(&old_owner->task->pi_lock);



-- Steve


PS.  I notice I have a large CC on this thread.  If anyone wants me to
take them off, just send me a private email, and I'll do that.

Re: 2.6.13-rc6-rt8

[Peter Bortas]

On 8/23/05, Ingo Molnar <mingo@elte.hu> wrote:
> 
> * Peter Bortas <peter@bortas.org> wrote:
> 
> > 2.6.13-rc6-rt8 fails to build with my configuration (attached):
> >
> > net/built-in.o: In function `ip_rt_init':
> > : undefined reference to `__you_cannot_kmalloc_that_much'
> > make[1]: *** [.tmp_vmlinux1] Error 1
> > make[1]: Leaving directory `/usr/src/linux-2.6.13-rc6'
> > make: *** [stamp-build] Error 2
> 
> ok, fixed the likely cause of this in -rt12. Could you check whether it
> builds for you now?
> 
>         Ingo

rc7-rt4 compiles fine now. 

(Sorry I missed your message when originally sent.)

-- 
Peter Bortas

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

OK, I'm looking for a second set of eyes (or third or more :-) to see if
there's a danger of a deadlock here.

The task->pi_lock is dependent on the order of locking, which I said was
proven to be true, otherwise there would be a deadlock.

L1->P1->L2->P2-L3->P3 

Where lock L1 is owned by P1 which is blocked on L2 and so on.

The problem comes when we have to deal with this BKL.  This lock is
special in that it can be grabbed and released and break this chain.
Mainly with semaphores, since BKL shouldn't be released when scheduling
where the old spin_locks were replaced.  But since we replaced
semaphores with mutexes, the BKL starts to change the rules.

(Ingo, my last patch is flawed since it only covers the case of
semaphores, so please don't apply it, yet).

Now we can have:

BKL->P1->L1->P2->BKL

Where there's a loop.  The good news is that this only happens when P1
is about to release the BKL, otherwise it would be a real deadlock in
the kernel and not with the PI.

My fear of a deadlock is in the __up code.  Where we have the locking
order of:  

P2->BKL->P1

Is there someplace that can cause this to deadlock?

To deadlock, we need another process on another CPU to lock in the
following order:

P1->L1->P2->BKL

Which I guess can happen and we would get a deadlock.  This task could
be holding P1 and L1 and be waiting on P2 while the __up could have P2
and BKL and be waiting on P1.

So would this solve the problem with this deadlock?  This would be in
pick_new_owner where the caller is the __up.  (the P2->BKL-P1 side) This
is not a patch, but I put +'s at the added code.


#ifdef CONFIG_SMP
 try_again:
#endif
	trace_special_pid(waiter->ti->task->pid, waiter->ti->task->prio, 0);

#ifdef ALL_TASKS_PI
	check_pi_list_present(lock, waiter, old_owner);
#endif
	new_owner = waiter->ti;
	/*
	 * The new owner is still blocked on this lock, so we
	 * must release the lock->wait_lock before grabing
	 * the new_owner lock.
	 */
	__raw_spin_unlock(&lock->wait_lock);
	__raw_spin_lock(&new_owner->task->pi_lock);
	__raw_spin_lock(&lock->wait_lock);
	/*
	 * In this split second of releasing the lock, a high priority
	 * process could have come along and blocked as well.
	 */
#ifdef CONFIG_SMP
	waiter = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, list);
	if (unlikely(waiter->ti != new_owner)) {
		__raw_spin_unlock(&new_owner->task->pi_lock);
		goto try_again;
	}
+	/*
+	 * Once again the BKL comes to play.  Since the BKL can be grabbed and released
+	 * out of the normal P1->L1->P2 order, there's a chance that someone has the
+	 * BKL owner's lock and is waiting on the new owner lock.
+	 */
+	if (unlikely(lock == &kernel_sem.lock)) {
+		if (!__raw_spin_trylock(&old_owner->task->pi_lock)) {
+			__raw_spin_unlock(&lock->wait_lock);
+			__raw_spin_unlock(&new_owner->task->pi_lock);
+			goto try_again;
+		}
+	} else
#endif
		__raw_spin_lock(&old_owner->task->pi_lock);


-- Steve

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

On Tue, 2005-08-30 at 11:00 -0400, Steven Rostedt wrote:
> OK, I'm looking for a second set of eyes (or third or more :-) to see if
> there's a danger of a deadlock here.

Unless someone sees a problem with the patch, here it is.  I noticed
that I unlocked the lock->wait_lock when I should not have.  (scares me
since I'm currently running the kernel with that bug!)

Ingo,  in the pi_setprio, I can stop the loop if the owner of the BKL
has a task lock_depth >= 0 correct?  This means that the owner is either
running, or is going to go to sleep.  If it was blocked on another task,
via a mutex, then this count would be -1.  Correct?  I may need to move
the setting of lock_depth = -1 to inside the protection of the task
pi_lock.

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_goliath/kernel/rt.c
===================================================================
--- linux_realtime_goliath/kernel/rt.c	(revision 310)
+++ linux_realtime_goliath/kernel/rt.c	(working copy)
@@ -1089,11 +1089,21 @@
 		__raw_spin_unlock(&new_owner->task->pi_lock);
 		goto try_again;
 	}
+	/*
+	 * Once again the BKL comes to play.  Since the BKL can be grabbed and released
+	 * out of the normal P1->L1->P2 order, there's a chance that someone has the
+	 * BKL owner's lock and is waiting on the new owner lock.
+	 */
+	if (unlikely(lock == &kernel_sem.lock)) {
+		if (!__raw_spin_trylock(&old_owner->task->pi_lock)) {
+			__raw_spin_unlock(&new_owner->task->pi_lock);
+			goto try_again;
+		}
+	} else
 #endif
+		__raw_spin_lock(&old_owner->task->pi_lock);
+
 	plist_del_init(&waiter->list, &lock->wait_list);
-
-	__raw_spin_lock(&old_owner->task->pi_lock);
-
 	plist_del(&waiter->pi_list, &old_owner->task->pi_waiters);
 	plist_init(&waiter->pi_list, waiter->ti->task->prio);
 

Re: [RFC] RT-patch update to remove the global pi_lock

[Steven Rostedt]

Ingo,

This patch contains my previous change as well as an update to fix the
race conditions that the BKL may hold.  It is against -rt2. 

The first part of the patch will stop the pi_setprio loop if the process
has a lock_depth greater than or equal to zero.  Since that would mean
that the process either is running, or is about to release the BKL.

I still kept the change from rt1 to rt2 but changed the comment.

I added the lock release logic in the __up code incase the BKL is
causing loops in the pi chain.

I'm currently runnig -rt2 with these changes as I type.

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux_realtime_goliath/kernel/rt.c
===================================================================
--- linux_realtime_goliath/kernel/rt.c	(revision 310)
+++ linux_realtime_goliath/kernel/rt.c	(working copy)
@@ -800,7 +800,24 @@
 #endif
 
 		mutex_setprio(p, prio);
-		if (!w)
+
+		/*
+		 * The BKL can really be a pain. It can happen where the
+		 * BKL is being held by one task that is just about to
+		 * block on another task that is waiting for the BKL.
+		 * This isn't a deadlock, since the BKL is released
+		 * when the task goes to sleep.  This also means that
+		 * all holders of the BKL are not blocked, or are just
+		 * about to be blocked.
+		 *
+		 * Another side-effect of this is that there's a small
+		 * window where the spinlocks are not held, and the blocked
+		 * process hasn't released the BKL.  So if we are going
+		 * to boost the owner of the BKL, stop after that,
+		 * since that owner is either running, or about to sleep
+		 * but don't go any further or we are in a loop.
+		 */
+		if (!w || unlikely(p->lock_depth >= 0))
 			break;
 		/*
 		 * If the task is blocked on a lock, and we just made
@@ -817,10 +834,9 @@
 		TRACE_BUG_ON_LOCKED(!lock);
 
 		/*
-		 * The BKL can really be a pain.  It can happen that the lock
-		 * we are blocked on is owned by a task that is waiting for
-		 * the BKL, and we own it.  So, if this is the BKL and we own
-		 * it, then end the loop here.
+		 * The current task that is blocking can also the one
+		 * holding the BKL, and blocking on a task that wants 
+		 * it.  So if it were to get this far, we would deadlock.
 		 */
 		if (unlikely(l == &kernel_sem.lock) && lock_owner(l) == current_thread_info()) {
 			/*
@@ -1089,11 +1105,21 @@
 		__raw_spin_unlock(&new_owner->task->pi_lock);
 		goto try_again;
 	}
+	/*
+	 * Once again the BKL comes to play.  Since the BKL can be grabbed and released
+	 * out of the normal P1->L1->P2 order, there's a chance that someone has the
+	 * BKL owner's lock and is waiting on the new owner lock.
+	 */
+	if (unlikely(lock == &kernel_sem.lock)) {
+		if (!__raw_spin_trylock(&old_owner->task->pi_lock)) {
+			__raw_spin_unlock(&new_owner->task->pi_lock);
+			goto try_again;
+		}
+	} else
 #endif
+		__raw_spin_lock(&old_owner->task->pi_lock);
+
 	plist_del_init(&waiter->list, &lock->wait_list);
-
-	__raw_spin_lock(&old_owner->task->pi_lock);
-
 	plist_del(&waiter->pi_list, &old_owner->task->pi_waiters);
 	plist_init(&waiter->pi_list, waiter->ti->task->prio);
 

[FYI] 2.6.13-rt3 and a nanosleep jitter test.

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 3116 bytes --]

Thomas,

I just was wondering how the HR Timers were in the latest -rtX patch and
wrote my own little jitter test using nanosleep.  Here's the results:

On vanilla 2.6.13-rc7-git1  (Yes I need to get over to 2.6.13)

# ./jitter
starting calibrate
finished calibrate: 2133.9060MHz 2133906034
time slept: 0.010000000 sec: 0 nsec: 10000000
max: 0.011997701
min: 0.011890522
avg: 0.011993485
greatest time over: 1997.701 usecs
never ran under (good!)
average time over: 1993.485 usecs

On 2.6.13-rt3:

# ./jitter
starting calibrate
finished calibrate: 2133.2960MHz 2133295991
time slept: 0.010000000 sec: 0 nsec: 10000000
max: 0.010034857
min: 0.010006309
avg: 0.010009213
greatest time over: 34.857 usecs
never ran under (good!)
average time over: 9.213 usecs


Not bad.  I then ran a clean kernel compile as root with a -j3 (this is
an 2x SMP machine), and tried the test again.

# ./jitter
starting calibrate
finished calibrate: 2133.3005MHz 2133300491
time slept: 0.010000000 sec: 0 nsec: 10000000
max: 0.010044836
min: 0.010014244
avg: 0.010030741
greatest time over: 44.836 usecs
never ran under (good!)
average time over: 30.741 usecs


Since the ticks per second is also calculated here, I ran it again using
the calibration of the first run (still running that make):

./jitter -c 2133295991
passed in calibrate: 2133.2960MHz 2133295991
time slept: 0.010000000 sec: 0 nsec: 10000000
max: 0.010051293
min: 0.010012155
avg: 0.010030237
greatest time over: 51.293 usecs
never ran under (good!)
average time over: 30.237 usecs

And once more using the calibration found by 2.6.13-rc7-git1 (still
running that make):

# ./jitter -c 2133906034
passed in calibrate: 2133.9060MHz 2133906034
time slept: 0.010000000 sec: 0 nsec: 10000000
max: 0.010058339
min: 0.010016418
avg: 0.010025571
greatest time over: 58.339 usecs
never ran under (good!)
average time over: 25.571 usecs


Some info on my machine:

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 10
model name      : AMD Athlon(tm) MP 2800+
stepping        : 0
cpu MHz         : 2133.286
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
bogomips        : 4259.84

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 10
model name      : AMD Athlon(tm) Processor
stepping        : 0
cpu MHz         : 2133.286
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
bogomips        : 4259.84

And attached is the jitter.c test. Must be run as root since it ups the
priority to the max.

Have fun,

-- Steve


[-- Attachment #2: jitter.c --]
[-- Type: text/x-csrc, Size: 2086 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <getopt.h>
#include <sched.h>
#include <unistd.h>

#define rdtsc(low,high) \
     __asm__ __volatile__("rdtsc" : "=a" (low), "=d" (high))

#define rdtscl(low) \
     __asm__ __volatile__("rdtsc" : "=a" (low) : : "edx")

#define rdtscll(val) \
     __asm__ __volatile__("rdtsc" : "=A" (val))

#define LOOPS 100.0

int main(int argc, char **argv)
{
	int c;
	unsigned long long start, end, tps = 0;
	double cal;
	double max=0,min=0,avg=0;
	int i;
	pid_t pid;
	struct sched_param sp;
	double sleeptime;
	unsigned long sec,nsec;

	sleeptime = 0.01;
	sec = sleeptime;
	nsec = (sleeptime-sec) * 1000000000;
	
	pid = getpid();

	memset(&sp,0,sizeof(sp));
	sp.sched_priority = sched_get_priority_max(SCHED_FIFO);
	sched_setscheduler(pid,SCHED_FIFO,&sp);

	while ((c=getopt(argc,argv,"c:")) != -1) {
		switch (c) {
			case 'c':
				tps = strtoll(optarg,NULL,10);
				break;
			default:
				exit(0);
		}
	}

	if (!tps) {
		printf("starting calibrate\n");
		rdtscll(start);
		sleep(8);
		rdtscll(end);

		tps = end - start;
		tps >>= 3;
		printf("finished calibrate: ");
	} else {
		printf("passed in calibrate: ");
	}
	cal = tps;
	cal /= 1000000.0;

	printf("%.4fMHz %llu\n",cal,tps);

	for (i=0; i < LOOPS; i++) {
		struct timespec req = { sec, nsec };
		unsigned long long t;
		rdtscll(start);
		nanosleep(&req,NULL);
		rdtscll(end);
		t = end - start;
		cal = t;
		cal /= (double)tps;
		if (!i || cal > max)
			max = cal;
		if (!i || cal < min)
			min = cal;
		avg += cal;
	}

	avg /= LOOPS;
	
	printf("time slept: %.9f sec: %lu nsec: %lu\n",sleeptime,sec,nsec);
	printf("max: %.9f\n",max);
	printf("min: %.9f\n",min);
	printf("avg: %.9f\n",avg);

	printf("greatest time over: %.3f usecs\n",(max-sleeptime)*1000000.0);
	if (min > sleeptime) {
		printf("never ran under (good!)\n");
	} else {
		printf("Ran under!!! (by %.9f) nsecs\n",sleeptime-min);
	}

	if (avg > sleeptime)
		printf("average time over: %.3f usecs\n",(avg-sleeptime)*1000000.0);
	else
		printf("average time ran under???\n");
	
	exit(0);
}

Re: [FYI] 2.6.13-rt3 and a nanosleep jitter test.

[Daniel Walker]

There is already a suite HRT of tests they include a nanosleep jitter
test with 8 or 9 other tests..

find them inside the hrt-support patch at http://high-res-timer.sf.net

Daniel

On Wed, 2005-08-31 at 11:01 -0400, Steven Rostedt wrote:
> Thomas,
> 
> I just was wondering how the HR Timers were in the latest -rtX patch and
> wrote my own little jitter test using nanosleep.  Here's the results:
> 
> On vanilla 2.6.13-rc7-git1  (Yes I need to get over to 2.6.13)
> 
> # ./jitter
> starting calibrate
> finished calibrate: 2133.9060MHz 2133906034
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.011997701
> min: 0.011890522
> avg: 0.011993485
> greatest time over: 1997.701 usecs
> never ran under (good!)
> average time over: 1993.485 usecs
> 
> On 2.6.13-rt3:
> 
> # ./jitter
> starting calibrate
> finished calibrate: 2133.2960MHz 2133295991
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.010034857
> min: 0.010006309
> avg: 0.010009213
> greatest time over: 34.857 usecs
> never ran under (good!)
> average time over: 9.213 usecs
> 
> 
> Not bad.  I then ran a clean kernel compile as root with a -j3 (this is
> an 2x SMP machine), and tried the test again.
> 
> # ./jitter
> starting calibrate
> finished calibrate: 2133.3005MHz 2133300491
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.010044836
> min: 0.010014244
> avg: 0.010030741
> greatest time over: 44.836 usecs
> never ran under (good!)
> average time over: 30.741 usecs
> 
> 
> Since the ticks per second is also calculated here, I ran it again using
> the calibration of the first run (still running that make):
> 
> ./jitter -c 2133295991
> passed in calibrate: 2133.2960MHz 2133295991
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.010051293
> min: 0.010012155
> avg: 0.010030237
> greatest time over: 51.293 usecs
> never ran under (good!)
> average time over: 30.237 usecs
> 
> And once more using the calibration found by 2.6.13-rc7-git1 (still
> running that make):
> 
> # ./jitter -c 2133906034
> passed in calibrate: 2133.9060MHz 2133906034
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.010058339
> min: 0.010016418
> avg: 0.010025571
> greatest time over: 58.339 usecs
> never ran under (good!)
> average time over: 25.571 usecs
> 
> 
> Some info on my machine:
> 
> $ cat /proc/cpuinfo
> processor       : 0
> vendor_id       : AuthenticAMD
> cpu family      : 6
> model           : 10
> model name      : AMD Athlon(tm) MP 2800+
> stepping        : 0
> cpu MHz         : 2133.286
> cache size      : 512 KB
> fdiv_bug        : no
> hlt_bug         : no
> f00f_bug        : no
> coma_bug        : no
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 1
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
> bogomips        : 4259.84
> 
> processor       : 1
> vendor_id       : AuthenticAMD
> cpu family      : 6
> model           : 10
> model name      : AMD Athlon(tm) Processor
> stepping        : 0
> cpu MHz         : 2133.286
> cache size      : 512 KB
> fdiv_bug        : no
> hlt_bug         : no
> f00f_bug        : no
> coma_bug        : no
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 1
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
> bogomips        : 4259.84
> 
> And attached is the jitter.c test. Must be run as root since it ups the
> priority to the max.
> 
> Have fun,
> 
> -- Steve
> 

Re: [FYI] 2.6.13-rt3 and a nanosleep jitter test.

[Daniel Walker]

Sorry, that's http://high-res-timers.sf.net/

Daniel

On Wed, 2005-08-31 at 11:01 -0400, Steven Rostedt wrote:
> Thomas,
> 
> I just was wondering how the HR Timers were in the latest -rtX patch and
> wrote my own little jitter test using nanosleep.  Here's the results:
> 
> On vanilla 2.6.13-rc7-git1  (Yes I need to get over to 2.6.13)
> 
> # ./jitter
> starting calibrate
> finished calibrate: 2133.9060MHz 2133906034
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.011997701
> min: 0.011890522
> avg: 0.011993485
> greatest time over: 1997.701 usecs
> never ran under (good!)
> average time over: 1993.485 usecs
> 
> On 2.6.13-rt3:
> 
> # ./jitter
> starting calibrate
> finished calibrate: 2133.2960MHz 2133295991
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.010034857
> min: 0.010006309
> avg: 0.010009213
> greatest time over: 34.857 usecs
> never ran under (good!)
> average time over: 9.213 usecs
> 
> 
> Not bad.  I then ran a clean kernel compile as root with a -j3 (this is
> an 2x SMP machine), and tried the test again.
> 
> # ./jitter
> starting calibrate
> finished calibrate: 2133.3005MHz 2133300491
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.010044836
> min: 0.010014244
> avg: 0.010030741
> greatest time over: 44.836 usecs
> never ran under (good!)
> average time over: 30.741 usecs
> 
> 
> Since the ticks per second is also calculated here, I ran it again using
> the calibration of the first run (still running that make):
> 
> ./jitter -c 2133295991
> passed in calibrate: 2133.2960MHz 2133295991
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.010051293
> min: 0.010012155
> avg: 0.010030237
> greatest time over: 51.293 usecs
> never ran under (good!)
> average time over: 30.237 usecs
> 
> And once more using the calibration found by 2.6.13-rc7-git1 (still
> running that make):
> 
> # ./jitter -c 2133906034
> passed in calibrate: 2133.9060MHz 2133906034
> time slept: 0.010000000 sec: 0 nsec: 10000000
> max: 0.010058339
> min: 0.010016418
> avg: 0.010025571
> greatest time over: 58.339 usecs
> never ran under (good!)
> average time over: 25.571 usecs
> 
> 
> Some info on my machine:
> 
> $ cat /proc/cpuinfo
> processor       : 0
> vendor_id       : AuthenticAMD
> cpu family      : 6
> model           : 10
> model name      : AMD Athlon(tm) MP 2800+
> stepping        : 0
> cpu MHz         : 2133.286
> cache size      : 512 KB
> fdiv_bug        : no
> hlt_bug         : no
> f00f_bug        : no
> coma_bug        : no
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 1
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
> bogomips        : 4259.84
> 
> processor       : 1
> vendor_id       : AuthenticAMD
> cpu family      : 6
> model           : 10
> model name      : AMD Athlon(tm) Processor
> stepping        : 0
> cpu MHz         : 2133.286
> cache size      : 512 KB
> fdiv_bug        : no
> hlt_bug         : no
> f00f_bug        : no
> coma_bug        : no
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 1
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
> bogomips        : 4259.84
> 
> And attached is the jitter.c test. Must be run as root since it ups the
> priority to the max.
> 
> Have fun,
> 
> -- Steve
> 

Re: [FYI] 2.6.13-rt3 and a nanosleep jitter test.

[Steven Rostedt]

On Wed, 2005-08-31 at 08:13 -0700, Daniel Walker wrote:
> Sorry, that's http://high-res-timers.sf.net/

Thanks,

But I always seem to prefer to rewrite the wheel than to use one that
already exists. ;-)  Probably explains why my cars are always in the
shop!

-- Steve

Re: [FYI] 2.6.13-rt3 and a nanosleep jitter test.

[Daniel Walker]

No problem .. There are some other tests in there though .. I've been
using them for stress testing .. Apparently you don't need HRT on to use
them either.

Daniel

On Wed, 2005-08-31 at 11:19 -0400, Steven Rostedt wrote:
> On Wed, 2005-08-31 at 08:13 -0700, Daniel Walker wrote:
> > Sorry, that's http://high-res-timers.sf.net/
> 
> Thanks,
> 
> But I always seem to prefer to rewrite the wheel than to use one that
> already exists. ;-)  Probably explains why my cars are always in the
> shop!
> 
> -- Steve
> 
> 

Re: [FYI] 2.6.13-rt3 and a nanosleep jitter test.

[Steven Rostedt]

On Wed, 2005-08-31 at 08:12 -0700, Daniel Walker wrote:
> There is already a suite HRT of tests they include a nanosleep jitter
> test with 8 or 9 other tests..
> 
> find them inside the hrt-support patch at http://high-res-timer.sf.net

Wow, they are hidden nicely :-)

I was looking for a tar ball or something. Not a patch.  Well, anyway I
took a look at it, and it is pretty much like mine. But I wanted to
eliminate any more system calls than I needed. Grant you, that my test
is not portable to anything other than ia86.  But it gives me the
results that I wanted.

Thanks,

-- Steve