[Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11989
Subject		: Suspend failure on NForce4-based boards due to chanes in stop_machine
Submitter	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
Date		: 2008-11-03 0:28 (7 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc
References	: http://marc.info/?l=linux-kernel&m=122567187604356&w=4

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Heiko Carstens]

On Sun, Nov 09, 2008 at 06:59:16PM +0100, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.27.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11989
> Subject		: Suspend failure on NForce4-based boards due to chanes in stop_machine
> Submitter	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
> Date		: 2008-11-03 0:28 (7 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc
> References	: http://marc.info/?l=linux-kernel&m=122567187604356&w=4

Hi Rafael,

could you provide more informations for this, please?

What is your kernel configuration?
Do you have any binary only modules (nvidia?) loaded?

Is it possible to recreate the bug by e.g. just doing something like

echo 0 > /sys/devices/system/cpu/cpu1/online

(or any other online cpu)? Or does it trigger any lockdep warnings?

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Rafael J. Wysocki]

On Monday, 10 of November 2008, Heiko Carstens wrote:
> On Sun, Nov 09, 2008 at 06:59:16PM +0100, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.27.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11989
> > Subject		: Suspend failure on NForce4-based boards due to chanes in stop_machine
> > Submitter	: Rafael J. Wysocki <rjw@sisk.pl>
> > Date		: 2008-11-03 0:28 (7 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc
> > References	: http://marc.info/?l=linux-kernel&m=122567187604356&w=4
> 
> Hi Rafael,

Hi,

> could you provide more informations for this, please?
> 
> What is your kernel configuration?

Available at: http://www.sisk.pl/kernel/debug/mainline/2.6.28-rc3/kitty-config

> Do you have any binary only modules (nvidia?) loaded?

No, I don't.

> Is it possible to recreate the bug by e.g. just doing something like
> 
> echo 0 > /sys/devices/system/cpu/cpu1/online

I haven't checked (yet), I'll do that later today and let you know.

> (or any other online cpu)? Or does it trigger any lockdep warnings?

Thanks,
Rafael

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Rafael J. Wysocki]

On Monday, 10 of November 2008, Rafael J. Wysocki wrote:
> On Monday, 10 of November 2008, Heiko Carstens wrote:
> > On Sun, Nov 09, 2008 at 06:59:16PM +0100, Rafael J. Wysocki wrote:
> > > This message has been generated automatically as a part of a report
> > > of recent regressions.
> > > 
> > > The following bug entry is on the current list of known regressions
> > > from 2.6.27.  Please verify if it still should be listed and let me know
> > > (either way).
> > > 
> > > 
> > > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11989
> > > Subject		: Suspend failure on NForce4-based boards due to chanes in stop_machine
> > > Submitter	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
> > > Date		: 2008-11-03 0:28 (7 days old)
> > > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc
> > > References	: http://marc.info/?l=linux-kernel&m=122567187604356&w=4
> > 
> > Hi Rafael,
> 
> Hi,
> 
> > could you provide more informations for this, please?
> > 
> > What is your kernel configuration?
> 
> Available at: http://www.sisk.pl/kernel/debug/mainline/2.6.28-rc3/kitty-config
> 
> > Do you have any binary only modules (nvidia?) loaded?
> 
> No, I don't.
> 
> > Is it possible to recreate the bug by e.g. just doing something like
> > 
> > echo 0 > /sys/devices/system/cpu/cpu1/online
> 
> I haven't checked (yet), I'll do that later today and let you know.
> 
> > (or any other online cpu)? Or does it trigger any lockdep warnings?

It cannot be reproduced with offlining CPU1 and it doesn't trigger any
warnings from lockdep.

However, it is reproducible by doing

# echo core > /sys/power/pm_test

and repeating

# echo disk > /sys/power/state

for a couple of times, in which case the last two lines printed to the console
before a (solid) hang are:

SMP alternatives: switching to SMP code
Booting processor 1 APIC 0x1 ip 0x6000

So, it evidently fails while re-enabling the non-boot CPU and not during
disabling it as I thought before.

With commit c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc reverted the issue is
not reproducible any more.

Thanks,
Rafael

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Ingo Molnar]

* Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> wrote:

> However, it is reproducible by doing
> 
> # echo core > /sys/power/pm_test
> 
> and repeating
> 
> # echo disk > /sys/power/state
> 
> for a couple of times, in which case the last two lines printed to the console
> before a (solid) hang are:
> 
> SMP alternatives: switching to SMP code
> Booting processor 1 APIC 0x1 ip 0x6000
> 
> So, it evidently fails while re-enabling the non-boot CPU and not 
> during disabling it as I thought before.
> 
> With commit c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc reverted the 
> issue is not reproducible any more.

[ Cc:-ed workqueue/locking/suspend-race-condition experts. ]

Seems like the new kernel/stop_machine.c logic has a race for the test 
sequence above. (Below is the bisected commit again, maybe the race is 
visible via email review as well.)

	Ingo

-------------->
From c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc Mon Sep 17 00:00:00 2001
From: Heiko Carstens <heiko.carstens-tA70FqPdS9bQT0dZR+AlfA@public.gmane.org>
Date: Mon, 13 Oct 2008 23:50:10 +0200
Subject: [PATCH] stop_machine: use workqueues instead of kernel threads

Convert stop_machine to a workqueue based approach. Instead of using kernel
threads for stop_machine we now use a an rt workqueue to synchronize all
cpus.
This has the advantage that all needed per cpu threads are already created
when stop_machine gets called. And therefore a call to stop_machine won't
fail anymore. This is needed for s390 which needs a mechanism to synchronize
all cpus without allocating any memory.
As Rusty pointed out free_module() needs a non-failing stop_machine interface
as well.

As a side effect the stop_machine code gets simplified.

Signed-off-by: Heiko Carstens <heiko.carstens-tA70FqPdS9bQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org>
---
 kernel/stop_machine.c |  111 ++++++++++++++++++-------------------------------
 1 files changed, 41 insertions(+), 70 deletions(-)

diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index af3c7ce..0e688c6 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -37,9 +37,13 @@ struct stop_machine_data {
 /* Like num_online_cpus(), but hotplug cpu uses us, so we need this. */
 static unsigned int num_threads;
 static atomic_t thread_ack;
-static struct completion finished;
 static DEFINE_MUTEX(lock);
 
+static struct workqueue_struct *stop_machine_wq;
+static struct stop_machine_data active, idle;
+static const cpumask_t *active_cpus;
+static void *stop_machine_work;
+
 static void set_state(enum stopmachine_state newstate)
 {
 	/* Reset ack counter. */
@@ -51,21 +55,25 @@ static void set_state(enum stopmachine_state newstate)
 /* Last one to ack a state moves to the next state. */
 static void ack_state(void)
 {
-	if (atomic_dec_and_test(&thread_ack)) {
-		/* If we're the last one to ack the EXIT, we're finished. */
-		if (state == STOPMACHINE_EXIT)
-			complete(&finished);
-		else
-			set_state(state + 1);
-	}
+	if (atomic_dec_and_test(&thread_ack))
+		set_state(state + 1);
 }
 
-/* This is the actual thread which stops the CPU.  It exits by itself rather
- * than waiting for kthread_stop(), because it's easier for hotplug CPU. */
-static int stop_cpu(struct stop_machine_data *smdata)
+/* This is the actual function which stops the CPU. It runs
+ * in the context of a dedicated stopmachine workqueue. */
+static void stop_cpu(struct work_struct *unused)
 {
 	enum stopmachine_state curstate = STOPMACHINE_NONE;
-
+	struct stop_machine_data *smdata = &idle;
+	int cpu = smp_processor_id();
+
+	if (!active_cpus) {
+		if (cpu == first_cpu(cpu_online_map))
+			smdata = &active;
+	} else {
+		if (cpu_isset(cpu, *active_cpus))
+			smdata = &active;
+	}
 	/* Simple state machine */
 	do {
 		/* Chill out and ensure we re-read stopmachine_state. */
@@ -90,7 +98,6 @@ static int stop_cpu(struct stop_machine_data *smdata)
 	} while (curstate != STOPMACHINE_EXIT);
 
 	local_irq_enable();
-	do_exit(0);
 }
 
 /* Callback for CPUs which aren't supposed to do anything. */
@@ -101,78 +108,34 @@ static int chill(void *unused)
 
 int __stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus)
 {
-	int i, err;
-	struct stop_machine_data active, idle;
-	struct task_struct **threads;
+	struct work_struct *sm_work;
+	int i;
 
+	/* Set up initial state. */
+	mutex_lock(&lock);
+	num_threads = num_online_cpus();
+	active_cpus = cpus;
 	active.fn = fn;
 	active.data = data;
 	active.fnret = 0;
 	idle.fn = chill;
 	idle.data = NULL;
 
-	/* This could be too big for stack on large machines. */
-	threads = kcalloc(NR_CPUS, sizeof(threads[0]), GFP_KERNEL);
-	if (!threads)
-		return -ENOMEM;
-
-	/* Set up initial state. */
-	mutex_lock(&lock);
-	init_completion(&finished);
-	num_threads = num_online_cpus();
 	set_state(STOPMACHINE_PREPARE);
 
-	for_each_online_cpu(i) {
-		struct stop_machine_data *smdata = &idle;
-		struct sched_param param = { .sched_priority = MAX_RT_PRIO-1 };
-
-		if (!cpus) {
-			if (i == first_cpu(cpu_online_map))
-				smdata = &active;
-		} else {
-			if (cpu_isset(i, *cpus))
-				smdata = &active;
-		}
-
-		threads[i] = kthread_create((void *)stop_cpu, smdata, "kstop%u",
-					    i);
-		if (IS_ERR(threads[i])) {
-			err = PTR_ERR(threads[i]);
-			threads[i] = NULL;
-			goto kill_threads;
-		}
-
-		/* Place it onto correct cpu. */
-		kthread_bind(threads[i], i);
-
-		/* Make it highest prio. */
-		if (sched_setscheduler_nocheck(threads[i], SCHED_FIFO, &param))
-			BUG();
-	}
-
-	/* We've created all the threads.  Wake them all: hold this CPU so one
+	/* Schedule the stop_cpu work on all cpus: hold this CPU so one
 	 * doesn't hit this CPU until we're ready. */
 	get_cpu();
-	for_each_online_cpu(i)
-		wake_up_process(threads[i]);
-
+	for_each_online_cpu(i) {
+		sm_work = percpu_ptr(stop_machine_work, i);
+		INIT_WORK(sm_work, stop_cpu);
+		queue_work_on(i, stop_machine_wq, sm_work);
+	}
 	/* This will release the thread on our CPU. */
 	put_cpu();
-	wait_for_completion(&finished);
+	flush_workqueue(stop_machine_wq);
 	mutex_unlock(&lock);
-
-	kfree(threads);
-
 	return active.fnret;
-
-kill_threads:
-	for_each_online_cpu(i)
-		if (threads[i])
-			kthread_stop(threads[i]);
-	mutex_unlock(&lock);
-
-	kfree(threads);
-	return err;
 }
 
 int stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus)
@@ -187,3 +150,11 @@ int stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus)
 	return ret;
 }
 EXPORT_SYMBOL_GPL(stop_machine);
+
+static int __init stop_machine_init(void)
+{
+	stop_machine_wq = create_rt_workqueue("kstop");
+	stop_machine_work = alloc_percpu(struct work_struct);
+	return 0;
+}
+early_initcall(stop_machine_init);

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Heiko Carstens]

On Tue, Nov 11, 2008 at 11:52:14AM +0100, Ingo Molnar wrote:
> 
> * Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> 
> > However, it is reproducible by doing
> > 
> > # echo core > /sys/power/pm_test
> > 
> > and repeating
> > 
> > # echo disk > /sys/power/state
> > 
> > for a couple of times, in which case the last two lines printed to the console
> > before a (solid) hang are:
> > 
> > SMP alternatives: switching to SMP code
> > Booting processor 1 APIC 0x1 ip 0x6000
> > 
> > So, it evidently fails while re-enabling the non-boot CPU and not 
> > during disabling it as I thought before.
> > 
> > With commit c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc reverted the 
> > issue is not reproducible any more.
> 
> [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]
> 
> Seems like the new kernel/stop_machine.c logic has a race for the test 
> sequence above. (Below is the bisected commit again, maybe the race is 
> visible via email review as well.)

FWIW, I tried to reproduce this on s390 and got the following:

A process that would do nothing but onlining/offlining cpus would get
stuck after a while:

 0 schedule+842 [0x342522]
 1 schedule_timeout+200 [0x342ec4]
 2 wait_for_common+362 [0x341fd6]
 3 wait_for_completion+54 [0x342146]
 4 __synchronize_sched+80 [0x81670]
 5 cpu_down+172 [0x33c030]
 6 store_online+96 [0x33c488]
 7 sysdev_store+52 [0x1bda84]
 8 sysfs_write_file+242 [0x1350ba]
 9 vfs_write+176 [0xd2028]
10 sys_write+82 [0xd21ea]
11 sysc_noemu+16 [0x269d8]

All cpus are in cpu_idle and no other task in state TASK_INTERRUPTIBLE
or TASK_UNINTERRUPTIBLE. However it would continue to work as soon as
I login into the system or generate a console interrupt.
I'm going to look into the dump and see if I can figure out what is
broken here.
Dunno if it is the same bug or something else.

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Heiko Carstens]

On Tue, Nov 11, 2008 at 12:31:34PM +0100, Heiko Carstens wrote:
> On Tue, Nov 11, 2008 at 11:52:14AM +0100, Ingo Molnar wrote:
> > 
> > * Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> > 
> > > However, it is reproducible by doing
> > > 
> > > # echo core > /sys/power/pm_test
> > > 
> > > and repeating
> > > 
> > > # echo disk > /sys/power/state
> > > 
> > > for a couple of times, in which case the last two lines printed to the console
> > > before a (solid) hang are:
> > > 
> > > SMP alternatives: switching to SMP code
> > > Booting processor 1 APIC 0x1 ip 0x6000
> > > 
> > > So, it evidently fails while re-enabling the non-boot CPU and not 
> > > during disabling it as I thought before.
> > > 
> > > With commit c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc reverted the 
> > > issue is not reproducible any more.
> > 
> > [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]
> > 
> > Seems like the new kernel/stop_machine.c logic has a race for the test 
> > sequence above. (Below is the bisected commit again, maybe the race is 
> > visible via email review as well.)
> 
> FWIW, I tried to reproduce this on s390 and got the following:
> 
> A process that would do nothing but onlining/offlining cpus would get
> stuck after a while:
> 
>  0 schedule+842 [0x342522]
>  1 schedule_timeout+200 [0x342ec4]
>  2 wait_for_common+362 [0x341fd6]
>  3 wait_for_completion+54 [0x342146]
>  4 __synchronize_sched+80 [0x81670]
>  5 cpu_down+172 [0x33c030]
>  6 store_online+96 [0x33c488]
>  7 sysdev_store+52 [0x1bda84]
>  8 sysfs_write_file+242 [0x1350ba]
>  9 vfs_write+176 [0xd2028]
> 10 sys_write+82 [0xd21ea]
> 11 sysc_noemu+16 [0x269d8]
> 
> All cpus are in cpu_idle and no other task in state TASK_INTERRUPTIBLE
> or TASK_UNINTERRUPTIBLE. However it would continue to work as soon as
> I login into the system or generate a console interrupt.
> I'm going to look into the dump and see if I can figure out what is
> broken here.
> Dunno if it is the same bug or something else.

[Cc:-ed Steven and Paul, since this backtrace seems to be RCU specific]

Steven, Paul, any idea what could cause the hang? I think I would
get lost in the RCU code...

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Ingo Molnar]

* Heiko Carstens <heiko.carstens-tA70FqPdS9bQT0dZR+AlfA@public.gmane.org> wrote:

> On Tue, Nov 11, 2008 at 12:31:34PM +0100, Heiko Carstens wrote:
> > On Tue, Nov 11, 2008 at 11:52:14AM +0100, Ingo Molnar wrote:
> > > 
> > > * Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> > > 
> > > > However, it is reproducible by doing
> > > > 
> > > > # echo core > /sys/power/pm_test
> > > > 
> > > > and repeating
> > > > 
> > > > # echo disk > /sys/power/state
> > > > 
> > > > for a couple of times, in which case the last two lines printed to the console
> > > > before a (solid) hang are:
> > > > 
> > > > SMP alternatives: switching to SMP code
> > > > Booting processor 1 APIC 0x1 ip 0x6000
> > > > 
> > > > So, it evidently fails while re-enabling the non-boot CPU and not 
> > > > during disabling it as I thought before.
> > > > 
> > > > With commit c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc reverted the 
> > > > issue is not reproducible any more.
> > > 
> > > [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]
> > > 
> > > Seems like the new kernel/stop_machine.c logic has a race for the test 
> > > sequence above. (Below is the bisected commit again, maybe the race is 
> > > visible via email review as well.)
> > 
> > FWIW, I tried to reproduce this on s390 and got the following:
> > 
> > A process that would do nothing but onlining/offlining cpus would get
> > stuck after a while:
> > 
> >  0 schedule+842 [0x342522]
> >  1 schedule_timeout+200 [0x342ec4]
> >  2 wait_for_common+362 [0x341fd6]
> >  3 wait_for_completion+54 [0x342146]
> >  4 __synchronize_sched+80 [0x81670]
> >  5 cpu_down+172 [0x33c030]
> >  6 store_online+96 [0x33c488]
> >  7 sysdev_store+52 [0x1bda84]
> >  8 sysfs_write_file+242 [0x1350ba]
> >  9 vfs_write+176 [0xd2028]
> > 10 sys_write+82 [0xd21ea]
> > 11 sysc_noemu+16 [0x269d8]
> > 
> > All cpus are in cpu_idle and no other task in state TASK_INTERRUPTIBLE
> > or TASK_UNINTERRUPTIBLE. However it would continue to work as soon as
> > I login into the system or generate a console interrupt.
> > I'm going to look into the dump and see if I can figure out what is
> > broken here.
> > Dunno if it is the same bug or something else.
> 
> [Cc:-ed Steven and Paul, since this backtrace seems to be RCU specific]
> 
> Steven, Paul, any idea what could cause the hang? I think I would
> get lost in the RCU code...

Cc:-ed Thomas - sometimes "RCU hangs" happen due to nohz confusion: 
because no timer IRQ happens so there's nothing to drive the RCU 
machinery.

	Ingo

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Vegard Nossum]

On Tue, Nov 11, 2008 at 11:52 AM, Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> wrote:
> [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]

Heh. I am not expert, but I looked at the code. The obvious suspicious
thing to see is the use of unpaired barriers? Maybe like this:

 47 static void set_state(enum stopmachine_state newstate)
48 {
49         /* Reset ack counter. */
50         atomic_set(&thread_ack, num_threads);
51         smp_wmb();

+ /* force ordering between thread_ack/state */

52         state = newstate;
53 }
54
55 /* Last one to ack a state moves to the next state. */
56 static void ack_state(void)
57 {
58         if (atomic_dec_and_test(&thread_ack))

Maybe
+ /* force ordering between thread_ack/state */
+ smp_rmb();
here?

59                 set_state(state + 1);
60 }
61

Or maybe I am wrong. But Documentation/memory-barriers.txt is rather
explicit on this point.


Vegard

-- 
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Vegard Nossum]

On Tue, Nov 11, 2008 at 2:36 PM, Vegard Nossum <vegard.nossum@gmail.com> wrote:
> On Tue, Nov 11, 2008 at 11:52 AM, Ingo Molnar <mingo@elte.hu> wrote:
>> [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]
>
> Heh. I am not expert, but I looked at the code. The obvious suspicious
> thing to see is the use of unpaired barriers? Maybe like this:

...

> 55 /* Last one to ack a state moves to the next state. */
> 56 static void ack_state(void)
> 57 {
> 58         if (atomic_dec_and_test(&thread_ack))
>
> Maybe
> + /* force ordering between thread_ack/state */
> + smp_rmb();
> here?

Oops, I am wrong (after a small investigation).

"1490 Any atomic operation that modifies some state in memory and
returns information
1491 about the state (old or new) implies an SMP-conditional general
memory barrier
1492 (smp_mb()) on each side of the actual operation (with the exception of
1493 explicit lock operations, described later).  These include:
1494
...
1503         atomic_dec_and_test();"

Won't fix the problem at hand, but maybe something like this would be
nice for future generations :-)

diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index 0e688c6..6796bb1 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -55,6 +55,7 @@ static void set_state(enum stopmachine_state newstate)
 /* Last one to ack a state moves to the next state. */
 static void ack_state(void)
 {
+       /* Implicit memory barrier; no smp_rmb() needed */
        if (atomic_dec_and_test(&thread_ack))
                set_state(state + 1);
 }


Vegard

-- 
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Peter Zijlstra]

On Tue, 2008-11-11 at 14:36 +0100, Vegard Nossum wrote:
> On Tue, Nov 11, 2008 at 11:52 AM, Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> wrote:
> > [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]
> 
> Heh. I am not expert, but I looked at the code. The obvious suspicious
> thing to see is the use of unpaired barriers? Maybe like this:
> 
>  47 static void set_state(enum stopmachine_state newstate)
> 48 {
> 49         /* Reset ack counter. */
> 50         atomic_set(&thread_ack, num_threads);
> 51         smp_wmb();
> 
> + /* force ordering between thread_ack/state */
> 
> 52         state = newstate;
> 53 }
> 54
> 55 /* Last one to ack a state moves to the next state. */
> 56 static void ack_state(void)
> 57 {
> 58         if (atomic_dec_and_test(&thread_ack))
> 
> Maybe
> + /* force ordering between thread_ack/state */
> + smp_rmb();
> here?

all atomic ops that have return values imply a full barrier, iirc

> 59                 set_state(state + 1);
> 60 }
> 61
> 
> Or maybe I am wrong. But Documentation/memory-barriers.txt is rather
> explicit on this point.
> 
> 
> Vegard
> 

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Paul E. McKenney]

On Tue, Nov 11, 2008 at 01:42:01PM +0100, Heiko Carstens wrote:
> On Tue, Nov 11, 2008 at 12:31:34PM +0100, Heiko Carstens wrote:
> > On Tue, Nov 11, 2008 at 11:52:14AM +0100, Ingo Molnar wrote:
> > > 
> > > * Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> > > 
> > > > However, it is reproducible by doing
> > > > 
> > > > # echo core > /sys/power/pm_test
> > > > 
> > > > and repeating
> > > > 
> > > > # echo disk > /sys/power/state
> > > > 
> > > > for a couple of times, in which case the last two lines printed to the console
> > > > before a (solid) hang are:
> > > > 
> > > > SMP alternatives: switching to SMP code
> > > > Booting processor 1 APIC 0x1 ip 0x6000
> > > > 
> > > > So, it evidently fails while re-enabling the non-boot CPU and not 
> > > > during disabling it as I thought before.
> > > > 
> > > > With commit c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc reverted the 
> > > > issue is not reproducible any more.
> > > 
> > > [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]
> > > 
> > > Seems like the new kernel/stop_machine.c logic has a race for the test 
> > > sequence above. (Below is the bisected commit again, maybe the race is 
> > > visible via email review as well.)
> > 
> > FWIW, I tried to reproduce this on s390 and got the following:
> > 
> > A process that would do nothing but onlining/offlining cpus would get
> > stuck after a while:
> > 
> >  0 schedule+842 [0x342522]
> >  1 schedule_timeout+200 [0x342ec4]
> >  2 wait_for_common+362 [0x341fd6]
> >  3 wait_for_completion+54 [0x342146]
> >  4 __synchronize_sched+80 [0x81670]
> >  5 cpu_down+172 [0x33c030]
> >  6 store_online+96 [0x33c488]
> >  7 sysdev_store+52 [0x1bda84]
> >  8 sysfs_write_file+242 [0x1350ba]
> >  9 vfs_write+176 [0xd2028]
> > 10 sys_write+82 [0xd21ea]
> > 11 sysc_noemu+16 [0x269d8]
> > 
> > All cpus are in cpu_idle and no other task in state TASK_INTERRUPTIBLE
> > or TASK_UNINTERRUPTIBLE. However it would continue to work as soon as
> > I login into the system or generate a console interrupt.
> > I'm going to look into the dump and see if I can figure out what is
> > broken here.
> > Dunno if it is the same bug or something else.
> 
> [Cc:-ed Steven and Paul, since this backtrace seems to be RCU specific]
> 
> Steven, Paul, any idea what could cause the hang? I think I would
> get lost in the RCU code...

Hello, Heiko,

Could you please apply the following debug patch (due to Jiangshan and
myself)?  Then you should be able to build with CONFIG_RCU_TRACE,
then mount debugfs after boot, for example, on /debug.  This will
create a /debug/rcu directory with three files, "rcucb", "rcu_data",
and "rcu_bh_data".  Since you are still able to log in, could you
please send the contents of these three files?

							Thanx, Paul

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Vegard Nossum]

On Tue, Nov 11, 2008 at 11:52 AM, Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> wrote:
> [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]
>
> Seems like the new kernel/stop_machine.c logic has a race for the test
> sequence above. (Below is the bisected commit again, maybe the race is
> visible via email review as well.)

I try again.

I think that the test for stop_machine_data in stop_cpu() should not
have been moved from __stop_machine(). Because now cpu_online_map may
change in-between calls to stop_cpu() (if the callback tries to
online/offline CPUs), and the end result may be different.

Maybe?


Vegard

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Heiko Carstens]

On Tue, Nov 11, 2008 at 06:35:05AM -0800, Paul E. McKenney wrote:
> > > A process that would do nothing but onlining/offlining cpus would get
> > > stuck after a while:
> > > 
> > >  0 schedule+842 [0x342522]
> > >  1 schedule_timeout+200 [0x342ec4]
> > >  2 wait_for_common+362 [0x341fd6]
> > >  3 wait_for_completion+54 [0x342146]
> > >  4 __synchronize_sched+80 [0x81670]
> > >  5 cpu_down+172 [0x33c030]
> > >  6 store_online+96 [0x33c488]
> > >  7 sysdev_store+52 [0x1bda84]
> > >  8 sysfs_write_file+242 [0x1350ba]
> > >  9 vfs_write+176 [0xd2028]
> > > 10 sys_write+82 [0xd21ea]
> > > 11 sysc_noemu+16 [0x269d8]
> > > 
> > > All cpus are in cpu_idle and no other task in state TASK_INTERRUPTIBLE
> > > or TASK_UNINTERRUPTIBLE. However it would continue to work as soon as
> > > I login into the system or generate a console interrupt.
> > > I'm going to look into the dump and see if I can figure out what is
> > > broken here.
> > > Dunno if it is the same bug or something else.
> > 
> > [Cc:-ed Steven and Paul, since this backtrace seems to be RCU specific]
> > 
> > Steven, Paul, any idea what could cause the hang? I think I would
> > get lost in the RCU code...
> 
> Hello, Heiko,
> 
> Could you please apply the following debug patch (due to Jiangshan and
> myself)?  Then you should be able to build with CONFIG_RCU_TRACE,
> then mount debugfs after boot, for example, on /debug.  This will
> create a /debug/rcu directory with three files, "rcucb", "rcu_data",
> and "rcu_bh_data".  Since you are still able to log in, could you
> please send the contents of these three files?

Hi Paul,

could you attach the patch please? :)

Does the patch also make sense if the system continues to work? That
is the machine isn't stalled anymore as soon as I log in.
On the other hand I do have a dump of the system and can look in
whatever data structures you want. If that helps.

Thanks,
Heiko

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Paul E. McKenney]

On Tue, Nov 11, 2008 at 06:35:05AM -0800, Paul E. McKenney wrote:
> On Tue, Nov 11, 2008 at 01:42:01PM +0100, Heiko Carstens wrote:
> > On Tue, Nov 11, 2008 at 12:31:34PM +0100, Heiko Carstens wrote:
> > > On Tue, Nov 11, 2008 at 11:52:14AM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Rafael J. Wysocki <rjw@sisk.pl> wrote:
> > > > 
> > > > > However, it is reproducible by doing
> > > > > 
> > > > > # echo core > /sys/power/pm_test
> > > > > 
> > > > > and repeating
> > > > > 
> > > > > # echo disk > /sys/power/state
> > > > > 
> > > > > for a couple of times, in which case the last two lines printed to the console
> > > > > before a (solid) hang are:
> > > > > 
> > > > > SMP alternatives: switching to SMP code
> > > > > Booting processor 1 APIC 0x1 ip 0x6000
> > > > > 
> > > > > So, it evidently fails while re-enabling the non-boot CPU and not 
> > > > > during disabling it as I thought before.
> > > > > 
> > > > > With commit c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc reverted the 
> > > > > issue is not reproducible any more.
> > > > 
> > > > [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]
> > > > 
> > > > Seems like the new kernel/stop_machine.c logic has a race for the test 
> > > > sequence above. (Below is the bisected commit again, maybe the race is 
> > > > visible via email review as well.)
> > > 
> > > FWIW, I tried to reproduce this on s390 and got the following:
> > > 
> > > A process that would do nothing but onlining/offlining cpus would get
> > > stuck after a while:
> > > 
> > >  0 schedule+842 [0x342522]
> > >  1 schedule_timeout+200 [0x342ec4]
> > >  2 wait_for_common+362 [0x341fd6]
> > >  3 wait_for_completion+54 [0x342146]
> > >  4 __synchronize_sched+80 [0x81670]
> > >  5 cpu_down+172 [0x33c030]
> > >  6 store_online+96 [0x33c488]
> > >  7 sysdev_store+52 [0x1bda84]
> > >  8 sysfs_write_file+242 [0x1350ba]
> > >  9 vfs_write+176 [0xd2028]
> > > 10 sys_write+82 [0xd21ea]
> > > 11 sysc_noemu+16 [0x269d8]
> > > 
> > > All cpus are in cpu_idle and no other task in state TASK_INTERRUPTIBLE
> > > or TASK_UNINTERRUPTIBLE. However it would continue to work as soon as
> > > I login into the system or generate a console interrupt.
> > > I'm going to look into the dump and see if I can figure out what is
> > > broken here.
> > > Dunno if it is the same bug or something else.
> > 
> > [Cc:-ed Steven and Paul, since this backtrace seems to be RCU specific]
> > 
> > Steven, Paul, any idea what could cause the hang? I think I would
> > get lost in the RCU code...
> 
> Hello, Heiko,
> 
> Could you please apply the following debug patch (due to Jiangshan and
> myself)?  Then you should be able to build with CONFIG_RCU_TRACE,
> then mount debugfs after boot, for example, on /debug.  This will
> create a /debug/rcu directory with three files, "rcucb", "rcu_data",
> and "rcu_bh_data".  Since you are still able to log in, could you
> please send the contents of these three files?
> 
> 							Thanx, Paul

This time with the patch actually attached...  Thanks to Peter Z.
for alerting me to my omission.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
---

diff --git a/include/linux/rcuclassic.h b/include/linux/rcuclassic.h
index 4ab8436..735f35a 100644
--- a/include/linux/rcuclassic.h
+++ b/include/linux/rcuclassic.h
@@ -54,6 +54,9 @@ struct rcu_ctrlblk {
 				 /* for current batch to proceed.        */
 } ____cacheline_internodealigned_in_smp;
 
+extern struct rcu_ctrlblk rcu_ctrlblk;
+extern struct rcu_ctrlblk rcu_bh_ctrlblk;
+
 /* Is batch a before batch b ? */
 static inline int rcu_batch_before(long a, long b)
 {
@@ -76,6 +79,7 @@ struct rcu_data {
 	long		quiescbatch;     /* Batch # for grace period */
 	int		passed_quiesc;	 /* User-mode/idle loop etc. */
 	int		qs_pending;	 /* core waits for quiesc state */
+	bool		beenonline;	 /* CPU online at least once */
 
 	/* 2) batch handling */
 	long  	       	batch;           /* Batch # for current RCU batch */
diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
index 9fdba03..ba32338 100644
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -68,7 +68,6 @@ config PREEMPT_RCU
 
 config RCU_TRACE
 	bool "Enable tracing for RCU - currently stats in debugfs"
-	depends on PREEMPT_RCU
 	select DEBUG_FS
 	default y
 	help
diff --git a/kernel/Makefile b/kernel/Makefile
index 4e1d7df..e0bfce7 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -77,6 +77,8 @@ obj-$(CONFIG_CLASSIC_RCU) += rcuclassic.o
 obj-$(CONFIG_PREEMPT_RCU) += rcupreempt.o
 ifeq ($(CONFIG_PREEMPT_RCU),y)
 obj-$(CONFIG_RCU_TRACE) += rcupreempt_trace.o
+else
+obj-$(CONFIG_RCU_TRACE) += rcuclassic_trace.o
 endif
 obj-$(CONFIG_RELAY) += relay.o
 obj-$(CONFIG_SYSCTL) += utsname_sysctl.o
diff --git a/kernel/rcuclassic.c b/kernel/rcuclassic.c
index aad93cd..06472fc 100644
--- a/kernel/rcuclassic.c
+++ b/kernel/rcuclassic.c
@@ -57,13 +57,13 @@ EXPORT_SYMBOL_GPL(rcu_lock_map);
 
 
 /* Definition for rcupdate control block. */
-static struct rcu_ctrlblk rcu_ctrlblk = {
+struct rcu_ctrlblk rcu_ctrlblk = {
 	.cur = -300,
 	.completed = -300,
 	.lock = __SPIN_LOCK_UNLOCKED(&rcu_ctrlblk.lock),
 	.cpumask = CPU_MASK_NONE,
 };
-static struct rcu_ctrlblk rcu_bh_ctrlblk = {
+struct rcu_ctrlblk rcu_bh_ctrlblk = {
 	.cur = -300,
 	.completed = -300,
 	.lock = __SPIN_LOCK_UNLOCKED(&rcu_bh_ctrlblk.lock),
@@ -564,6 +564,7 @@ static void rcu_init_percpu_data(int cpu, struct rcu_ctrlblk *rcp,
 	rdp->donetail = &rdp->donelist;
 	rdp->quiescbatch = rcp->completed;
 	rdp->qs_pending = 0;
+	rdp->beenonline = 1;
 	rdp->cpu = cpu;
 	rdp->blimit = blimit;
 }
diff --git a/kernel/rcuclassic_trace.c b/kernel/rcuclassic_trace.c
new file mode 100644
index 0000000..b719048
--- /dev/null
+++ b/kernel/rcuclassic_trace.c
@@ -0,0 +1,198 @@
+/*
+ * Read-Copy Update tracing for classic implementation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Copyright IBM Corporation, 2008
+ *
+ * Updated to use seqfile by Lai Jiangshan.
+ *
+ * Papers:  http://www.rdrop.com/users/paulmck/RCU
+ *
+ * For detailed explanation of Read-Copy Update mechanism see -
+ * 		Documentation/RCU
+ *
+ */
+#include <linux/rcupdate.h>
+#include <linux/module.h>
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+
+/* Print out rcu_data structures using seqfile facility. */
+
+static struct rcu_data *get_rcu_data_bh(int cpu)
+{
+	return &per_cpu(rcu_bh_data, cpu);
+}
+
+static struct rcu_data *get_rcu_data(int cpu)
+{
+	return &per_cpu(rcu_data, cpu);
+}
+
+static int show_rcu_data(struct seq_file *m, void *v)
+{
+	struct rcu_data *rdp = v;
+
+	if (!rdp->beenonline)
+		return 0;
+
+	seq_printf(m, "processor\t: %d", rdp->cpu);
+	if (cpu_is_offline(rdp->cpu))
+		seq_puts(m, "!\n");
+	else
+		seq_puts(m, "\n");
+	seq_printf(m, "quiescbatch\t: %ld\n", rdp->quiescbatch);
+	seq_printf(m, "batch\t\t: %ld\n", rdp->batch);
+	seq_printf(m, "passed_quiesc\t: %d\n", rdp->passed_quiesc);
+	seq_printf(m, "qs_pending\t: %d\n", rdp->qs_pending);
+	seq_printf(m, "qlen\t\t: %ld\n", rdp->qlen);
+	seq_printf(m, "blimit\t\t: %ld\n", rdp->blimit);
+	seq_puts(m, "\n");
+	return 0;
+}
+
+static void *c_start(struct seq_file *m, loff_t *pos)
+{
+	typedef struct rcu_data *(*get_data_func)(int);
+
+	if (*pos == 0)  /* just in case, cpu 0 is not the first */
+		*pos = first_cpu(cpu_possible_map);
+	else
+		*pos = next_cpu_nr(*pos - 1, cpu_possible_map);
+	if ((*pos) < nr_cpu_ids)
+		return ((get_data_func)m->private)(*pos);
+	return NULL;
+}
+
+static void *c_next(struct seq_file *m, void *v, loff_t *pos)
+{
+	(*pos)++;
+	return c_start(m, pos);
+}
+
+static void c_stop(struct seq_file *m, void *v)
+{
+}
+
+const struct seq_operations rcu_data_seq_op = {
+	.start	= c_start,
+	.next	= c_next,
+	.stop	= c_stop,
+	.show	= show_rcu_data,
+};
+
+static int rcu_data_open(struct inode *inode, struct file *file)
+{
+	int ret = seq_open(file, &rcu_data_seq_op);
+
+	if (ret)
+		return ret;
+	((struct seq_file *)file->private_data)->private = inode->i_private;
+	return 0;
+}
+
+static const struct file_operations rcu_data_fops = {
+	.owner		= THIS_MODULE,
+	.open		= rcu_data_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+};
+
+/* Print out rcu_ctrlblk structures using seqfile facility. */
+
+static void print_one_rcu_ctrlblk(struct seq_file *m, struct rcu_ctrlblk *rcp)
+{
+	seq_printf(m, "cur=%ld  completed=%ld   next_pending=%d  s=%d\n\t",
+		   rcp->cur, rcp->completed, rcp->next_pending, rcp->signaled);
+	seq_cpumask(m, &rcp->cpumask);
+	seq_puts(m, "\n");
+}
+
+static int show_rcucb(struct seq_file *m, void *unused)
+{
+	seq_puts(m, "rcu: ");
+	print_one_rcu_ctrlblk(m, &rcu_ctrlblk);
+	seq_puts(m, "rcu_bh: ");
+	print_one_rcu_ctrlblk(m, &rcu_bh_ctrlblk);
+	seq_puts(m, "online: ");
+	seq_cpumask(m, &cpu_online_map);
+	seq_puts(m, "\n");
+	return 0;
+}
+
+static int rcucb_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, show_rcucb, NULL);
+}
+
+static struct file_operations rcucb_fops = {
+	.owner		= THIS_MODULE,
+	.open		= rcucb_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
+static struct dentry *rcudir, *rcu_bh_data_file, *rcu_data_file, *rcucb_file;
+
+static int __init rcuclassic_trace_init(void)
+{
+	rcudir = debugfs_create_dir("rcu", NULL);
+	if (!rcudir)
+		goto out;
+
+	rcu_bh_data_file = debugfs_create_file("rcu_bh_data", 0444, rcudir,
+					       get_rcu_data_bh, &rcu_data_fops);
+	if (!rcu_bh_data_file)
+		goto out_rcudir;
+
+	rcu_data_file = debugfs_create_file("rcu_data", 0444, rcudir,
+					    get_rcu_data, &rcu_data_fops);
+	if (!rcu_data_file)
+		goto out_rcudata_bh_file;
+
+	rcucb_file = debugfs_create_file("rcucb", 0444, rcudir,
+					 NULL, &rcucb_fops);
+	if (!rcucb_file)
+		goto out_rcudata_file;
+	return 0;
+
+out_rcudata_file:
+	debugfs_remove(rcu_data_file);
+out_rcudata_bh_file:
+	debugfs_remove(rcu_bh_data_file);
+out_rcudir:
+	debugfs_remove(rcudir);
+out:
+	return 1;
+}
+
+static void __exit rcuclassic_trace_cleanup(void)
+{
+	debugfs_remove(rcucb_file);
+	debugfs_remove(rcu_data_file);
+	debugfs_remove(rcu_bh_data_file);
+	debugfs_remove(rcudir);
+}
+
+module_init(rcuclassic_trace_init);
+module_exit(rcuclassic_trace_cleanup);
+
+MODULE_AUTHOR("Paul E. McKenney");
+MODULE_DESCRIPTION("Read-Copy Update tracing for classic implementation");
+MODULE_LICENSE("GPL");
+

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Dmitry Adamushko]

2008/11/11 Vegard Nossum <vegard.nossum-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> On Tue, Nov 11, 2008 at 11:52 AM, Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> wrote:
>> [ Cc:-ed workqueue/locking/suspend-race-condition experts. ]
>>
>> Seems like the new kernel/stop_machine.c logic has a race for the test
>> sequence above. (Below is the bisected commit again, maybe the race is
>> visible via email review as well.)
>
> I try again.
>
> I think that the test for stop_machine_data in stop_cpu() should not
> have been moved from __stop_machine().

Do you mean the following test?

        if (!active_cpus) {
                if (cpu == first_cpu(cpu_online_map))
                        smdata = &active;
        } else {
                if (cpu_isset(cpu, *active_cpus))
                        smdata = &active;
        }

> Because now cpu_online_map may
> change in-between calls to stop_cpu() (if the callback tries to
> online/offline CPUs), and the end result may be different.

take_cpu_down() may not run earlier than stop_cpu() on all the cpus
have completed the STOPMACHINE_DISABLE_IRQ step, iow. "state ==
STOPMACHINE_RUN". By that moment, 'smdata' has been set up on all
cpus... if this is the case you had in mind.


>
> Maybe?
>
>
> Vegard
>


-- 
Best regards,
Dmitry Adamushko

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Heiko Carstens]

> > Could you please apply the following debug patch (due to Jiangshan and
> > myself)?  Then you should be able to build with CONFIG_RCU_TRACE,
> > then mount debugfs after boot, for example, on /debug.  This will
> > create a /debug/rcu directory with three files, "rcucb", "rcu_data",
> > and "rcu_bh_data".  Since you are still able to log in, could you
> > please send the contents of these three files?
> > 
> > 							Thanx, Paul
> 
> This time with the patch actually attached...  Thanks to Peter Z.
> for alerting me to my omission.

Well, your patch doesn't apply on git head. However I used preemptible
RCU instead and had tracing enabled.

This is the output of the three files after it stalled (and continued,
because I caused an interrupt by sending a network packet) twice:

[root@h0545001 rcu]# cat rcuctrs 
CPU last cur F M
  1    0   0 1 1
  3    0   0 1 1
  4    0   0 0 0
  5    0   0 0 1
  6    0   0 0 0
ggp = 1640, state = waitack

[root@h0545001 rcu]# cat rcugp 
oldggp=1652  newggp=1655

[root@h0545001 rcu]# cat rcustats 
na=33948 nl=3 wa=33945 wl=0 da=33945 dl=0 dr=33945 di=0
1=0 e1=0 i1=1674 ie1=4 g1=1670 a1=1920 ae1=251 a2=1669
z1=1669 ze1=0 z2=1669 m1=4411 me1=2742 m2=1669

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Paul E. McKenney]

On Tue, Nov 11, 2008 at 04:01:32PM +0100, Heiko Carstens wrote:
> On Tue, Nov 11, 2008 at 06:35:05AM -0800, Paul E. McKenney wrote:
> > > > A process that would do nothing but onlining/offlining cpus would get
> > > > stuck after a while:
> > > > 
> > > >  0 schedule+842 [0x342522]
> > > >  1 schedule_timeout+200 [0x342ec4]
> > > >  2 wait_for_common+362 [0x341fd6]
> > > >  3 wait_for_completion+54 [0x342146]
> > > >  4 __synchronize_sched+80 [0x81670]
> > > >  5 cpu_down+172 [0x33c030]
> > > >  6 store_online+96 [0x33c488]
> > > >  7 sysdev_store+52 [0x1bda84]
> > > >  8 sysfs_write_file+242 [0x1350ba]
> > > >  9 vfs_write+176 [0xd2028]
> > > > 10 sys_write+82 [0xd21ea]
> > > > 11 sysc_noemu+16 [0x269d8]
> > > > 
> > > > All cpus are in cpu_idle and no other task in state TASK_INTERRUPTIBLE
> > > > or TASK_UNINTERRUPTIBLE. However it would continue to work as soon as
> > > > I login into the system or generate a console interrupt.
> > > > I'm going to look into the dump and see if I can figure out what is
> > > > broken here.
> > > > Dunno if it is the same bug or something else.
> > > 
> > > [Cc:-ed Steven and Paul, since this backtrace seems to be RCU specific]
> > > 
> > > Steven, Paul, any idea what could cause the hang? I think I would
> > > get lost in the RCU code...
> > 
> > Hello, Heiko,
> > 
> > Could you please apply the following debug patch (due to Jiangshan and
> > myself)?  Then you should be able to build with CONFIG_RCU_TRACE,
> > then mount debugfs after boot, for example, on /debug.  This will
> > create a /debug/rcu directory with three files, "rcucb", "rcu_data",
> > and "rcu_bh_data".  Since you are still able to log in, could you
> > please send the contents of these three files?
> 
> Hi Paul,
> 
> could you attach the patch please? :)

Peter Z. beat you to it.  ;-)

See previous email.

> Does the patch also make sense if the system continues to work? That
> is the machine isn't stalled anymore as soon as I log in.
> On the other hand I do have a dump of the system and can look in
> whatever data structures you want. If that helps.

Ah!

I would like to see the value of rcu_ctrlblk.cpumask and also the value
of cpu_online_map.  One guess would be that rcu_ctrlblk.cpumask has a
bit set that is -not- set in cpu_online_map, which would indicate that
RCU was incorrectly waiting on an offline CPU.

On the other hand, if all the bits set in rcu_ctrlblk.cpumask are also
set in cpu_online_map, then could you please dump out the instances of
the rcu_data per-CPU variable that correspond to the bits set in
rcu_ctrlblk.cpumask?

Finally, if no bits are set in rcu_ctrlblk.cpumask, the question would
be "why isn't the synchronize_sched() waking up?"

BTW, I am assuming that you have the same config as Raphael, in other
words, that you are running Classic RCU rather than preemptable RCU.

The point of the patch is that it allows you to see this info by catting
out the /debug/rcu files, at least assuming that the system is healthy
enough to allow you to cat files.  But if you already have a crash dump...

							Thanx, Paul

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Oleg Nesterov]

On 11/11, Vegard Nossum wrote:
>
> I think that the test for stop_machine_data in stop_cpu() should not
> have been moved from __stop_machine(). Because now cpu_online_map may
> change in-between calls to stop_cpu() (if the callback tries to
> online/offline CPUs), and the end result may be different.

I don't think this is possible, the callback must not be called unless
all threads ack (at least) the STOPMACHINE_PREPARE state.


Off-topic question, __stop_machine() does:
	
	/* Schedule the stop_cpu work on all cpus: hold this CPU so one
	 * doesn't hit this CPU until we're ready. */
	get_cpu();
	for_each_online_cpu(i) {
		sm_work = percpu_ptr(stop_machine_work, i);
		INIT_WORK(sm_work, stop_cpu);
		queue_work_on(i, stop_machine_wq, sm_work);
	}
	/* This will release the thread on our CPU. */
	put_cpu();

Don't we actually need preempt_disable/preempt_enable instead of
get/put cpu? (yes, there the same currently). We don't care about
the CPU we are running on, and it can't go away until we queue all
works. But we must ensure that stop_cpu() on the same CPU can't
preempt us, right?

Oleg.

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Paul E. McKenney]

On Tue, Nov 11, 2008 at 05:14:01PM +0100, Heiko Carstens wrote:
> > > Could you please apply the following debug patch (due to Jiangshan and
> > > myself)?  Then you should be able to build with CONFIG_RCU_TRACE,
> > > then mount debugfs after boot, for example, on /debug.  This will
> > > create a /debug/rcu directory with three files, "rcucb", "rcu_data",
> > > and "rcu_bh_data".  Since you are still able to log in, could you
> > > please send the contents of these three files?
> > > 
> > > 							Thanx, Paul
> > 
> > This time with the patch actually attached...  Thanks to Peter Z.
> > for alerting me to my omission.
> 
> Well, your patch doesn't apply on git head. However I used preemptible
> RCU instead and had tracing enabled.

Were you using preemptible RCU earlier as well?  Raphael was using
classic RCU.  Don't get me wrong, all problems need fixing, just trying
to make sure I understand where the problems are occurring.

> This is the output of the three files after it stalled (and continued,
> because I caused an interrupt by sending a network packet) twice:
> 
> [root@h0545001 rcu]# cat rcuctrs 
> CPU last cur F M
>   1    0   0 1 1
>   3    0   0 1 1
>   4    0   0 0 0
>   5    0   0 0 1
>   6    0   0 0 0
> ggp = 1640, state = waitack
> 
> [root@h0545001 rcu]# cat rcugp 
> oldggp=1652  newggp=1655
> 
> [root@h0545001 rcu]# cat rcustats 
> na=33948 nl=3 wa=33945 wl=0 da=33945 dl=0 dr=33945 di=0
> 1=0 e1=0 i1=1674 ie1=4 g1=1670 a1=1920 ae1=251 a2=1669
> z1=1669 ze1=0 z2=1669 m1=4411 me1=2742 m2=1669

This hang also involved synchronize_sched()?  Or synchronize_rcu()?

The reason I ask is that the above stats are for the synchronize_rcu()
rather than synchronize_sched().

							Thanx, Paul

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Paul E. McKenney]

On Tue, Nov 11, 2008 at 08:45:23AM -0800, Paul E. McKenney wrote:
> On Tue, Nov 11, 2008 at 05:14:01PM +0100, Heiko Carstens wrote:
> > > > Could you please apply the following debug patch (due to Jiangshan and
> > > > myself)?  Then you should be able to build with CONFIG_RCU_TRACE,
> > > > then mount debugfs after boot, for example, on /debug.  This will
> > > > create a /debug/rcu directory with three files, "rcucb", "rcu_data",
> > > > and "rcu_bh_data".  Since you are still able to log in, could you
> > > > please send the contents of these three files?
> > > > 
> > > > 							Thanx, Paul
> > > 
> > > This time with the patch actually attached...  Thanks to Peter Z.
> > > for alerting me to my omission.
> > 
> > Well, your patch doesn't apply on git head. However I used preemptible
> > RCU instead and had tracing enabled.
> 
> Were you using preemptible RCU earlier as well?  Raphael was using
> classic RCU.  Don't get me wrong, all problems need fixing, just trying
> to make sure I understand where the problems are occurring.

And here is a version of the patch rebased to linux-2.6 git head.

This adds tracing to classic RCU.

Signed-off-by: Paul E. McKenney <paulmck-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
Signed-off-by: Lai Jiangshan <laijs-BthXqXjhjHXQFUHtdCDX3A@public.gmane.org>
---

 include/linux/rcuclassic.h |    4 
 kernel/Kconfig.preempt     |    1 
 kernel/Makefile            |    2 
 kernel/rcuclassic.c        |    5 -
 kernel/rcuclassic_trace.c  |  198 +++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 207 insertions(+), 3 deletions(-)

diff --git a/include/linux/rcuclassic.h b/include/linux/rcuclassic.h
index 5f89b62..ce183a8 100644
--- a/include/linux/rcuclassic.h
+++ b/include/linux/rcuclassic.h
@@ -63,6 +63,9 @@ struct rcu_ctrlblk {
 				 /* for current batch to proceed.        */
 } ____cacheline_internodealigned_in_smp;
 
+extern struct rcu_ctrlblk rcu_ctrlblk;
+extern struct rcu_ctrlblk rcu_bh_ctrlblk;
+
 /* Is batch a before batch b ? */
 static inline int rcu_batch_before(long a, long b)
 {
@@ -81,6 +84,7 @@ struct rcu_data {
 	long		quiescbatch;     /* Batch # for grace period */
 	int		passed_quiesc;	 /* User-mode/idle loop etc. */
 	int		qs_pending;	 /* core waits for quiesc state */
+	bool		beenonline;	 /* CPU online at least once */
 
 	/* 2) batch handling */
 	/*
diff --git a/kernel/Kconfig.preempt b/kernel/Kconfig.preempt
index 9fdba03..ba32338 100644
--- a/kernel/Kconfig.preempt
+++ b/kernel/Kconfig.preempt
@@ -68,7 +68,6 @@ config PREEMPT_RCU
 
 config RCU_TRACE
 	bool "Enable tracing for RCU - currently stats in debugfs"
-	depends on PREEMPT_RCU
 	select DEBUG_FS
 	default y
 	help
diff --git a/kernel/Makefile b/kernel/Makefile
index 9a3ec66..9771050 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -79,6 +79,8 @@ obj-$(CONFIG_CLASSIC_RCU) += rcuclassic.o
 obj-$(CONFIG_PREEMPT_RCU) += rcupreempt.o
 ifeq ($(CONFIG_PREEMPT_RCU),y)
 obj-$(CONFIG_RCU_TRACE) += rcupreempt_trace.o
+else
+obj-$(CONFIG_RCU_TRACE) += rcuclassic_trace.o
 endif
 obj-$(CONFIG_RELAY) += relay.o
 obj-$(CONFIG_SYSCTL) += utsname_sysctl.o
diff --git a/kernel/rcuclassic.c b/kernel/rcuclassic.c
index 37f72e5..54bd23b 100644
--- a/kernel/rcuclassic.c
+++ b/kernel/rcuclassic.c
@@ -58,14 +58,14 @@ EXPORT_SYMBOL_GPL(rcu_lock_map);
 
 
 /* Definition for rcupdate control block. */
-static struct rcu_ctrlblk rcu_ctrlblk = {
+struct rcu_ctrlblk rcu_ctrlblk = {
 	.cur = -300,
 	.completed = -300,
 	.pending = -300,
 	.lock = __SPIN_LOCK_UNLOCKED(&rcu_ctrlblk.lock),
 	.cpumask = CPU_MASK_NONE,
 };
-static struct rcu_ctrlblk rcu_bh_ctrlblk = {
+struct rcu_ctrlblk rcu_bh_ctrlblk = {
 	.cur = -300,
 	.completed = -300,
 	.pending = -300,
@@ -725,6 +725,7 @@ static void rcu_init_percpu_data(int cpu, struct rcu_ctrlblk *rcp,
 	rdp->donetail = &rdp->donelist;
 	rdp->quiescbatch = rcp->completed;
 	rdp->qs_pending = 0;
+	rdp->beenonline = 1;
 	rdp->cpu = cpu;
 	rdp->blimit = blimit;
 	spin_unlock_irqrestore(&rcp->lock, flags);
diff --git a/kernel/rcuclassic_trace.c b/kernel/rcuclassic_trace.c
new file mode 100644
index 0000000..612170c
--- /dev/null
+++ b/kernel/rcuclassic_trace.c
@@ -0,0 +1,198 @@
+/*
+ * Read-Copy Update tracing for classic implementation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Copyright IBM Corporation, 2008
+ *
+ * Updated to use seqfile by Lai Jiangshan.
+ *
+ * Papers:  http://www.rdrop.com/users/paulmck/RCU
+ *
+ * For detailed explanation of Read-Copy Update mechanism see -
+ * 		Documentation/RCU
+ *
+ */
+#include <linux/rcupdate.h>
+#include <linux/module.h>
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+
+/* Print out rcu_data structures using seqfile facility. */
+
+static struct rcu_data *get_rcu_data_bh(int cpu)
+{
+	return &per_cpu(rcu_bh_data, cpu);
+}
+
+static struct rcu_data *get_rcu_data(int cpu)
+{
+	return &per_cpu(rcu_data, cpu);
+}
+
+static int show_rcu_data(struct seq_file *m, void *v)
+{
+	struct rcu_data *rdp = v;
+
+	if (!rdp->beenonline)
+		return 0;
+
+	seq_printf(m, "processor\t: %d", rdp->cpu);
+	if (cpu_is_offline(rdp->cpu))
+		seq_puts(m, "!\n");
+	else
+		seq_puts(m, "\n");
+	seq_printf(m, "quiescbatch\t: %ld\n", rdp->quiescbatch);
+	seq_printf(m, "batch\t\t: %ld\n", rdp->batch);
+	seq_printf(m, "passed_quiesc\t: %d\n", rdp->passed_quiesc);
+	seq_printf(m, "qs_pending\t: %d\n", rdp->qs_pending);
+	seq_printf(m, "qlen\t\t: %ld\n", rdp->qlen);
+	seq_printf(m, "blimit\t\t: %ld\n", rdp->blimit);
+	seq_puts(m, "\n");
+	return 0;
+}
+
+static void *c_start(struct seq_file *m, loff_t *pos)
+{
+	typedef struct rcu_data *(*get_data_func)(int);
+
+	if (*pos == 0)  /* just in case, cpu 0 is not the first */
+		*pos = first_cpu(cpu_possible_map);
+	else
+		*pos = next_cpu_nr(*pos - 1, cpu_possible_map);
+	if ((*pos) < nr_cpu_ids)
+		return ((get_data_func)m->private)(*pos);
+	return NULL;
+}
+
+static void *c_next(struct seq_file *m, void *v, loff_t *pos)
+{
+	(*pos)++;
+	return c_start(m, pos);
+}
+
+static void c_stop(struct seq_file *m, void *v)
+{
+}
+
+const struct seq_operations rcu_data_seq_op = {
+	.start	= c_start,
+	.next	= c_next,
+	.stop	= c_stop,
+	.show	= show_rcu_data,
+};
+
+static int rcu_data_open(struct inode *inode, struct file *file)
+{
+	int ret = seq_open(file, &rcu_data_seq_op);
+
+	if (ret)
+		return ret;
+	((struct seq_file *)file->private_data)->private = inode->i_private;
+	return 0;
+}
+
+static const struct file_operations rcu_data_fops = {
+	.owner		= THIS_MODULE,
+	.open		= rcu_data_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+};
+
+/* Print out rcu_ctrlblk structures using seqfile facility. */
+
+static void print_one_rcu_ctrlblk(struct seq_file *m, struct rcu_ctrlblk *rcp)
+{
+	seq_printf(m, "cur=%ld  completed=%ld   pending=%d  s=%d\n\t",
+		   rcp->cur, rcp->completed, rcp->pending, rcp->signaled);
+	seq_cpumask(m, &rcp->cpumask);
+	seq_puts(m, "\n");
+}
+
+static int show_rcucb(struct seq_file *m, void *unused)
+{
+	seq_puts(m, "rcu: ");
+	print_one_rcu_ctrlblk(m, &rcu_ctrlblk);
+	seq_puts(m, "rcu_bh: ");
+	print_one_rcu_ctrlblk(m, &rcu_bh_ctrlblk);
+	seq_puts(m, "online: ");
+	seq_cpumask(m, &cpu_online_map);
+	seq_puts(m, "\n");
+	return 0;
+}
+
+static int rcucb_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, show_rcucb, NULL);
+}
+
+static struct file_operations rcucb_fops = {
+	.owner		= THIS_MODULE,
+	.open		= rcucb_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
+static struct dentry *rcudir, *rcu_bh_data_file, *rcu_data_file, *rcucb_file;
+
+static int __init rcuclassic_trace_init(void)
+{
+	rcudir = debugfs_create_dir("rcu", NULL);
+	if (!rcudir)
+		goto out;
+
+	rcu_bh_data_file = debugfs_create_file("rcu_bh_data", 0444, rcudir,
+					       get_rcu_data_bh, &rcu_data_fops);
+	if (!rcu_bh_data_file)
+		goto out_rcudir;
+
+	rcu_data_file = debugfs_create_file("rcu_data", 0444, rcudir,
+					    get_rcu_data, &rcu_data_fops);
+	if (!rcu_data_file)
+		goto out_rcudata_bh_file;
+
+	rcucb_file = debugfs_create_file("rcucb", 0444, rcudir,
+					 NULL, &rcucb_fops);
+	if (!rcucb_file)
+		goto out_rcudata_file;
+	return 0;
+
+out_rcudata_file:
+	debugfs_remove(rcu_data_file);
+out_rcudata_bh_file:
+	debugfs_remove(rcu_bh_data_file);
+out_rcudir:
+	debugfs_remove(rcudir);
+out:
+	return 1;
+}
+
+static void __exit rcuclassic_trace_cleanup(void)
+{
+	debugfs_remove(rcucb_file);
+	debugfs_remove(rcu_data_file);
+	debugfs_remove(rcu_bh_data_file);
+	debugfs_remove(rcudir);
+}
+
+module_init(rcuclassic_trace_init);
+module_exit(rcuclassic_trace_cleanup);
+
+MODULE_AUTHOR("Paul E. McKenney");
+MODULE_DESCRIPTION("Read-Copy Update tracing for classic implementation");
+MODULE_LICENSE("GPL");
+

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Dmitry Adamushko]

2008/11/10 Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>:
> On Monday, 10 of November 2008, Rafael J. Wysocki wrote:
>> On Monday, 10 of November 2008, Heiko Carstens wrote:
>> > On Sun, Nov 09, 2008 at 06:59:16PM +0100, Rafael J. Wysocki wrote:
>> > > This message has been generated automatically as a part of a report
>> > > of recent regressions.
>> > >
>> > > The following bug entry is on the current list of known regressions
>> > > from 2.6.27.  Please verify if it still should be listed and let me know
>> > > (either way).
>> > >
>> > >
>> > > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=11989
>> > > Subject           : Suspend failure on NForce4-based boards due to chanes in stop_machine
>> > > Submitter : Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
>> > > Date              : 2008-11-03 0:28 (7 days old)
>> > > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc
>> > > References        : http://marc.info/?l=linux-kernel&m=122567187604356&w=4
>> >
>> > Hi Rafael,
>>
>> Hi,
>>
>> > could you provide more informations for this, please?
>> >
>> > What is your kernel configuration?
>>
>> Available at: http://www.sisk.pl/kernel/debug/mainline/2.6.28-rc3/kitty-config
>>
>> > Do you have any binary only modules (nvidia?) loaded?
>>
>> No, I don't.
>>
>> > Is it possible to recreate the bug by e.g. just doing something like
>> >
>> > echo 0 > /sys/devices/system/cpu/cpu1/online
>>
>> I haven't checked (yet), I'll do that later today and let you know.
>>
>> > (or any other online cpu)? Or does it trigger any lockdep warnings?
>
> It cannot be reproduced with offlining CPU1 and it doesn't trigger any
> warnings from lockdep.
>
> However, it is reproducible by doing
>
> # echo core > /sys/power/pm_test
>
> and repeating
>
> # echo disk > /sys/power/state
>
> for a couple of times, in which case the last two lines printed to the console
> before a (solid) hang are:
>
> SMP alternatives: switching to SMP code
> Booting processor 1 APIC 0x1 ip 0x6000
>
> So, it evidently fails while re-enabling the non-boot CPU and not during
> disabling it as I thought before.

Can you also provide the full log including the messages when a system
goes down please?

At first glance, "Botting processor..." as the last message looks
strange in this context.
So either wakeup_secondary_cpu()'s completion failed for some reason
(say, due to some kind of a problem that took place while disabling
non-boot cpus... I'm purely speculating here so far) or the printk's
output was not complete.

Perhaps, redoing the test with pr_debug() in arch/x86/kernel/smpboot.c
enabled would shed more light...


-- 
Best regards,
Dmitry Adamushko

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Rafael J. Wysocki]

On Tuesday, 11 of November 2008, Dmitry Adamushko wrote:
> 2008/11/10 Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>:
> > On Monday, 10 of November 2008, Rafael J. Wysocki wrote:
> >> On Monday, 10 of November 2008, Heiko Carstens wrote:
> >> > On Sun, Nov 09, 2008 at 06:59:16PM +0100, Rafael J. Wysocki wrote:
> >> > > This message has been generated automatically as a part of a report
> >> > > of recent regressions.
> >> > >
> >> > > The following bug entry is on the current list of known regressions
> >> > > from 2.6.27.  Please verify if it still should be listed and let me know
> >> > > (either way).
> >> > >
> >> > >
> >> > > Bug-Entry : http://bugzilla.kernel.org/show_bug.cgi?id=11989
> >> > > Subject           : Suspend failure on NForce4-based boards due to chanes in stop_machine
> >> > > Submitter : Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
> >> > > Date              : 2008-11-03 0:28 (7 days old)
> >> > > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc
> >> > > References        : http://marc.info/?l=linux-kernel&m=122567187604356&w=4
> >> >
> >> > Hi Rafael,
> >>
> >> Hi,
> >>
> >> > could you provide more informations for this, please?
> >> >
> >> > What is your kernel configuration?
> >>
> >> Available at: http://www.sisk.pl/kernel/debug/mainline/2.6.28-rc3/kitty-config
> >>
> >> > Do you have any binary only modules (nvidia?) loaded?
> >>
> >> No, I don't.
> >>
> >> > Is it possible to recreate the bug by e.g. just doing something like
> >> >
> >> > echo 0 > /sys/devices/system/cpu/cpu1/online
> >>
> >> I haven't checked (yet), I'll do that later today and let you know.
> >>
> >> > (or any other online cpu)? Or does it trigger any lockdep warnings?
> >
> > It cannot be reproduced with offlining CPU1 and it doesn't trigger any
> > warnings from lockdep.
> >
> > However, it is reproducible by doing
> >
> > # echo core > /sys/power/pm_test
> >
> > and repeating
> >
> > # echo disk > /sys/power/state
> >
> > for a couple of times, in which case the last two lines printed to the console
> > before a (solid) hang are:
> >
> > SMP alternatives: switching to SMP code
> > Booting processor 1 APIC 0x1 ip 0x6000
> >
> > So, it evidently fails while re-enabling the non-boot CPU and not during
> > disabling it as I thought before.
> 
> Can you also provide the full log including the messages when a system
> goes down please?
> 
> At first glance, "Botting processor..." as the last message looks
> strange in this context.
> So either wakeup_secondary_cpu()'s completion failed for some reason
> (say, due to some kind of a problem that took place while disabling
> non-boot cpus... I'm purely speculating here so far) or the printk's
> output was not complete.
>
> Perhaps, redoing the test with pr_debug() in arch/x86/kernel/smpboot.c
> enabled would shed more light...

Will do tomorrow.

Thanks,
Rafael

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Rusty Russell]

On Wednesday 12 November 2008 03:01:18 Oleg Nesterov wrote:
> On 11/11, Vegard Nossum wrote:
> > I think that the test for stop_machine_data in stop_cpu() should not
> > have been moved from __stop_machine(). Because now cpu_online_map may
> > change in-between calls to stop_cpu() (if the callback tries to
> > online/offline CPUs), and the end result may be different.
>
> I don't think this is possible, the callback must not be called unless
> all threads ack (at least) the STOPMACHINE_PREPARE state.
>
>
> Off-topic question, __stop_machine() does:
>
> 	/* Schedule the stop_cpu work on all cpus: hold this CPU so one
> 	 * doesn't hit this CPU until we're ready. */
> 	get_cpu();
> 	for_each_online_cpu(i) {
> 		sm_work = percpu_ptr(stop_machine_work, i);
> 		INIT_WORK(sm_work, stop_cpu);
> 		queue_work_on(i, stop_machine_wq, sm_work);
> 	}
> 	/* This will release the thread on our CPU. */
> 	put_cpu();
>
> Don't we actually need preempt_disable/preempt_enable instead of
> get/put cpu? (yes, there the same currently). We don't care about
> the CPU we are running on, and it can't go away until we queue all
> works. But we must ensure that stop_cpu() on the same CPU can't
> preempt us, right?

A subtle distinction, but yes.  It used to be true before the recent changes, 
where we manually did "this" cpu.

Cheers,
Rusty.

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Rusty Russell]

On Tuesday 11 November 2008 21:22:14 Ingo Molnar wrote:
> * Rafael J. Wysocki <rjw@sisk.pl> wrote:
> > So, it evidently fails while re-enabling the non-boot CPU and not
> > during disabling it as I thought before.

(Resend, due to HTML version previously)

But what is calling stop_machine in that path?

There *is* a race, but I don't think it could cause this (we should make a
copy of active.fnret inside the lock before returning it).

Two patches: one fixes that race, the next adds debugging spew.

stop_machine: fix race with return value

We should not access active.fnret outside the lock; in theory the next
stop_machine could overwrite it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
---
 kernel/stop_machine.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff -r d7c9a15da615 kernel/stop_machine.c
--- a/kernel/stop_machine.c	Mon Nov 10 09:47:45 2008 +1100
+++ b/kernel/stop_machine.c	Tue Nov 11 23:19:47 2008 +1030
@@ -112,7 +112,7 @@
 int __stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus)
 {
 	struct work_struct *sm_work;
-	int i;
+	int i, ret;
 
 	/* Set up initial state. */
 	mutex_lock(&lock);
@@ -137,8 +137,9 @@
 	/* This will release the thread on our CPU. */
 	put_cpu();
 	flush_workqueue(stop_machine_wq);
+	ret = active.fnret;
 	mutex_unlock(&lock);
-	return active.fnret;
+	return ret;
 }
 
 int stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus)
===
diff -r fe7dd39b1cff kernel/stop_machine.c
--- a/kernel/stop_machine.c	Wed Nov 12 14:07:18 2008 +1030
+++ b/kernel/stop_machine.c	Wed Nov 12 14:09:08 2008 +1030
@@ -89,6 +89,8 @@
 			case STOPMACHINE_RUN:
 				/* On multiple CPUs only a single error code
 				 * is needed to tell that something failed. */
+				printk("stop_machine: %i running %p\n",
+				       smp_processor_id(), smdata->fn);
 				err = smdata->fn(smdata->data);
 				if (err)
 					smdata->fnret = err;
@@ -106,6 +108,7 @@
 /* Callback for CPUs which aren't supposed to do anything. */
 static int chill(void *unused)
 {
+	printk("stop_machine: %i chilling\n", smp_processor_id());
 	return 0;
 }
 
@@ -126,17 +129,23 @@
 
 	set_state(STOPMACHINE_PREPARE);
 
+	printk("stop_machine: running on %i cpus:\n", num_threads);
+	dump_stack();
+
 	/* Schedule the stop_cpu work on all cpus: hold this CPU so one
 	 * doesn't hit this CPU until we're ready. */
 	get_cpu();
 	for_each_online_cpu(i) {
+		printk("stop_machine: setting up cpu %i\n", i);
 		sm_work = percpu_ptr(stop_machine_work, i);
 		INIT_WORK(sm_work, stop_cpu);
 		queue_work_on(i, stop_machine_wq, sm_work);
 	}
 	/* This will release the thread on our CPU. */
+	printk("stop_machine: releasing CPU %i\n", smp_processor_id());
 	put_cpu();
 	flush_workqueue(stop_machine_wq);
+	printk("stop_machine: done\n");
 	ret = active.fnret;
 	mutex_unlock(&lock);
 	return ret;
\0

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Heiko Carstens]

On Tue, Nov 11, 2008 at 09:34:51AM -0800, Paul E. McKenney wrote:
> On Tue, Nov 11, 2008 at 08:45:23AM -0800, Paul E. McKenney wrote:
> > On Tue, Nov 11, 2008 at 05:14:01PM +0100, Heiko Carstens wrote:
> > > > > Could you please apply the following debug patch (due to Jiangshan and
> > > > > myself)?  Then you should be able to build with CONFIG_RCU_TRACE,
> > > > > then mount debugfs after boot, for example, on /debug.  This will
> > > > > create a /debug/rcu directory with three files, "rcucb", "rcu_data",
> > > > > and "rcu_bh_data".  Since you are still able to log in, could you
> > > > > please send the contents of these three files?
> > > > > 
> > > > > 							Thanx, Paul
> > > > 
> > > > This time with the patch actually attached...  Thanks to Peter Z.
> > > > for alerting me to my omission.
> > > 
> > > Well, your patch doesn't apply on git head. However I used preemptible
> > > RCU instead and had tracing enabled.
> > 
> > Were you using preemptible RCU earlier as well?  Raphael was using
> > classic RCU.  Don't get me wrong, all problems need fixing, just trying
> > to make sure I understand where the problems are occurring.

Indeed, my fault. I just try to reproduce a cpu hotplug bug with classic RCU
and cpu hotplug stress test, but no luck so far.

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Paul E. McKenney]

On Wed, Nov 12, 2008 at 10:05:08AM +0100, Heiko Carstens wrote:
> On Tue, Nov 11, 2008 at 09:34:51AM -0800, Paul E. McKenney wrote:
> > On Tue, Nov 11, 2008 at 08:45:23AM -0800, Paul E. McKenney wrote:
> > > On Tue, Nov 11, 2008 at 05:14:01PM +0100, Heiko Carstens wrote:
> > > > > > Could you please apply the following debug patch (due to Jiangshan and
> > > > > > myself)?  Then you should be able to build with CONFIG_RCU_TRACE,
> > > > > > then mount debugfs after boot, for example, on /debug.  This will
> > > > > > create a /debug/rcu directory with three files, "rcucb", "rcu_data",
> > > > > > and "rcu_bh_data".  Since you are still able to log in, could you
> > > > > > please send the contents of these three files?
> > > > > > 
> > > > > > 							Thanx, Paul
> > > > > 
> > > > > This time with the patch actually attached...  Thanks to Peter Z.
> > > > > for alerting me to my omission.
> > > > 
> > > > Well, your patch doesn't apply on git head. However I used preemptible
> > > > RCU instead and had tracing enabled.
> > > 
> > > Were you using preemptible RCU earlier as well?  Raphael was using
> > > classic RCU.  Don't get me wrong, all problems need fixing, just trying
> > > to make sure I understand where the problems are occurring.
> 
> Indeed, my fault. I just try to reproduce a cpu hotplug bug with classic RCU
> and cpu hotplug stress test, but no luck so far.

OK, then my next step will be to send Rafael an updated version of
my hierarchical RCU, which is more robust than classic RCU against
online/offline stress tests.  On the machines I have access to, anyway.  ;-)

Then I will look at preemptable RCU, which undoubtably needs some of the
same help that I have been giving to hierarchical RCU.  Manfred thus
wins the clairvoyance award!

							Thanx, Paul

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Heiko Carstens]

On Wed, Nov 12, 2008 at 08:03:49AM -0800, Paul E. McKenney wrote:
> On Wed, Nov 12, 2008 at 10:05:08AM +0100, Heiko Carstens wrote:
> > On Tue, Nov 11, 2008 at 09:34:51AM -0800, Paul E. McKenney wrote:
> > > > Were you using preemptible RCU earlier as well?  Raphael was using
> > > > classic RCU.  Don't get me wrong, all problems need fixing, just trying
> > > > to make sure I understand where the problems are occurring.
> > 
> > Indeed, my fault. I just try to reproduce a cpu hotplug bug with classic RCU
> > and cpu hotplug stress test, but no luck so far.
> 
> OK, then my next step will be to send Rafael an updated version of
> my hierarchical RCU, which is more robust than classic RCU against
> online/offline stress tests.  On the machines I have access to, anyway.  ;-)
> 
> Then I will look at preemptable RCU, which undoubtably needs some of the
> same help that I have been giving to hierarchical RCU.  Manfred thus
> wins the clairvoyance award!

Well, I tried all day long to reproduce a cpu hotplug/stop_machine hang
with classic RCU and a kernel configuration that is as close as possible
to Raphael's configuration, but it just continues to work without a bug.

One of the machines is a virtual machine with 8 virtual cpus mapped on
two real cpus. The real cpus are again shared with other guests. So I end
up with cpu steal times of 50-90%. That should have revealed races in the
stop_machine code, considering that thousands of cpu hotplug operations
happened.

I let these test machines running over night. Maybe something happens...
but at a first glance it looks more like the reworked stop_machine code
triggers a different bug that already is present. Hmmm...

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Paul E. McKenney]

On Wed, Nov 12, 2008 at 05:51:18PM +0100, Heiko Carstens wrote:
> On Wed, Nov 12, 2008 at 08:03:49AM -0800, Paul E. McKenney wrote:
> > On Wed, Nov 12, 2008 at 10:05:08AM +0100, Heiko Carstens wrote:
> > > On Tue, Nov 11, 2008 at 09:34:51AM -0800, Paul E. McKenney wrote:
> > > > > Were you using preemptible RCU earlier as well?  Raphael was using
> > > > > classic RCU.  Don't get me wrong, all problems need fixing, just trying
> > > > > to make sure I understand where the problems are occurring.
> > > 
> > > Indeed, my fault. I just try to reproduce a cpu hotplug bug with classic RCU
> > > and cpu hotplug stress test, but no luck so far.
> > 
> > OK, then my next step will be to send Rafael an updated version of
> > my hierarchical RCU, which is more robust than classic RCU against
> > online/offline stress tests.  On the machines I have access to, anyway.  ;-)
> > 
> > Then I will look at preemptable RCU, which undoubtably needs some of the
> > same help that I have been giving to hierarchical RCU.  Manfred thus
> > wins the clairvoyance award!
> 
> Well, I tried all day long to reproduce a cpu hotplug/stop_machine hang
> with classic RCU and a kernel configuration that is as close as possible
> to Raphael's configuration, but it just continues to work without a bug.
> 
> One of the machines is a virtual machine with 8 virtual cpus mapped on
> two real cpus. The real cpus are again shared with other guests. So I end
> up with cpu steal times of 50-90%. That should have revealed races in the
> stop_machine code, considering that thousands of cpu hotplug operations
> happened.
> 
> I let these test machines running over night. Maybe something happens...
> but at a first glance it looks more like the reworked stop_machine code
> triggers a different bug that already is present. Hmmm...

I can make Classic RCU break in 2.6.28-rc3, but I need a 128-CPU machine to
break it.  ;-)

							Thanx, Paul

Re: [Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Rafael J. Wysocki]

On Wednesday, 12 of November 2008, Rusty Russell wrote:
> On Tuesday 11 November 2008 21:22:14 Ingo Molnar wrote:
> > * Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> > > So, it evidently fails while re-enabling the non-boot CPU and not
> > > during disabling it as I thought before.
> 
> (Resend, due to HTML version previously)
> 
> But what is calling stop_machine in that path?
> 
> There *is* a race, but I don't think it could cause this (we should make a
> copy of active.fnret inside the lock before returning it).

Still, that seems to be the case.

> Two patches: one fixes that race, the next adds debugging spew.
> 
> stop_machine: fix race with return value

With this patch applied (reproduced below for clarity) the problem is not
reproducible any more.

Care to push it upstream ASAP?

Thanks,
Rafael

---
stop_machine: fix race with return value

We should not access active.fnret outside the lock; in theory the next
stop_machine could overwrite it.

Signed-off-by: Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org>
---
 kernel/stop_machine.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff -r d7c9a15da615 kernel/stop_machine.c
--- a/kernel/stop_machine.c	Mon Nov 10 09:47:45 2008 +1100
+++ b/kernel/stop_machine.c	Tue Nov 11 23:19:47 2008 +1030
@@ -112,7 +112,7 @@
 int __stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus)
 {
 	struct work_struct *sm_work;
-	int i;
+	int i, ret;
 
 	/* Set up initial state. */
 	mutex_lock(&lock);
@@ -137,8 +137,9 @@
 	/* This will release the thread on our CPU. */
 	put_cpu();
 	flush_workqueue(stop_machine_wq);
+	ret = active.fnret;
 	mutex_unlock(&lock);
-	return active.fnret;
+	return ret;
 }
 
 int stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus)

2.6.28-rc5: Reported regressions from 2.6.27

[Rafael J. Wysocki]

[NOTES:
 (1) Due to some recent controversies, in future I will only use 'Handled-By'
     tags for people who actually submit a patch or provide substantial help to
     the reporter (like advising him which commits to revert etc.).
 (2) We have almost as many regressions with patches as unresolved ones and
     the majority of these patches have not been merged for over two weeks
     (for almost three weeks in many cases and for over a _month_ in one case).
     IMO, this is insane.]

This message contains a list of some regressions from 2.6.27, for which there
are no fixes in the mainline I know of.  If any of them have been fixed already,
please let me know.

If you know of any other unresolved regressions from 2.6.27, please let me know
either and I'll add them to the list.  Also, please let me know if any of the
entries below are invalid.

Each entry from the list will be sent additionally in an automatic reply to
this message with CCs to the people involved in reporting and handling the
issue.


Listed regressions statistics:

  Date          Total  Pending  Unresolved
  ----------------------------------------
  2008-11-16       89       32          18
  2008-11-09       73       40          27
  2008-11-02       55       41          29
  2008-10-25       26       25          20


Unresolved regressions
----------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12049
Subject		: Oops in acpi_system_wakeup_device_seq_show
Submitter	: Bruno Prémont <bonbons-ud5FBsm0p/xEiooADzr8i9i2O/JbrIOy@public.gmane.org>
Date		: 2008-11-16 13:04 (1 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0794469da3f7b2093575cbdfc1108308dd3641ce
References	: http://marc.info/?l=linux-kernel&m=122684070215520&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12041
Subject		: 2.6.28-rc4 mem_cgroup_charge_common panic
Submitter	: Badari Pulavarty <pbadari-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
Date		: 2008-11-10 21:43 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=122635343008147&w=4
Handled-By	: KAMEZAWA Hiroyuki <kamezawa.hiroyu-+CUm20s59erQFUHtdCDX3A@public.gmane.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12040
Subject		: iwlagn driver segfault in 2.6.28-rc3
Submitter	: Lukas Hejtmanek <xhejtman-8qz54MUs51PtwjQa/ONI9g@public.gmane.org>
Date		: 2008-11-11 14:34 (6 days old)
References	: http://marc.info/?l=linux-kernel&m=122641417331593&w=4
Handled-By	: reinette chatre <reinette.chatre-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12034
Subject		: snd-hda-intel on Realtek ALC268 chip shows only Master volume (for playback)
Submitter	: Sergey <azure-IGUVrPOATOfsG83rWm+8vg@public.gmane.org>
Date		: 2008-11-15 04:20 (2 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12031
Subject		: DRM enabled kernel hangs hard on resume on x60
Submitter	: Jens Axboe <jens.axboe-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Date		: 2008-11-12 18:42 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=122651551216820&w=4
Handled-By	: Jesse Barnes <jbarnes-Y1mF5jBUw70BENJcbMCuUQ@public.gmane.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12028
Subject		: i915 DRM is broken in 2.6.28-rc4
Submitter	: Adam Tkac <vonsch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2008-11-14 01:50 (3 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12019
Subject		: Major increase in the number of wakeups from C3
Submitter	: Mircea Gherzan <mgherzan-0aWM6n0cdD1zL9IiBdi9GK9KmVUGpFAF@public.gmane.org>
Date		: 2008-11-13 07:02 (4 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11996
Subject		: Tracing framework regression in 2.6.28-rc3
Submitter	: Pekka Paalanen <pq-X3B1VOXEql0@public.gmane.org>
Date		: 2008-11-09 10:13 (8 days old)
References	: http://marc.info/?l=linux-kernel&m=122624392229317&w=4
Handled-By	: Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11970
Subject		: gettimeofday return a old time in mmbench
Submitter	: alexs <alex.shi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2008-11-06 23:57 (11 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=99ebcf8285df28f32fd2d1c19a7166e70f00309c
Handled-By	: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
		  Thomas Gleixner <tglx-hfZtesqFncYOwBW4kG4KsQ@public.gmane.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11965
Subject		: regression introduced by - timers: fix itimer/many thread hang
Submitter	: Doug Chapman <doug.chapman-VXdhtT5mjnY@public.gmane.org>
Date		: 2008-11-06 11:03 (11 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=f06febc96ba8e0af80bcc3eaec0a109e88275fac
References	: http://marc.info/?l=linux-kernel&m=122596943416648&w=4
Handled-By	: Frank Mayhar <fmayhar-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
		  Peter Zijlstra <peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
		  Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11958
Subject		: [2.6.27.x =&gt; 2.6.28-rc3] Xorg crash with xf86MapVidMem error
Submitter	: Tomasz Chmielewski <tch-Nem3ZqsbT/g@public.gmane.org>
Date		: 2008-11-05 05:37 (12 days old)


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11947
Subject		: 2.6.28-rc VC switching with Intel graphics broken
Submitter	: Romano Giannetti <romano.giannetti-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2008-11-03 12:10 (14 days old)
Handled-By	: Jesse Barnes <jbarnes-Y1mF5jBUw70BENJcbMCuUQ@public.gmane.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11913
Subject		: USB/INPUT: slab error in cache_alloc_debugcheck_after(): double free?
Submitter	: Helge Deller <deller-Mmb7MZpHnFY@public.gmane.org>
Date		: 2008-10-30 23:11 (18 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=cb8f488c33539f096580e202f5438a809195008f
References	: http://marc.info/?l=linux-kernel&m=122540833301394&w=4
Handled-By	: Jiri Kosina <jkosina-AlSwsSmVLrQ@public.gmane.org>
		  Jiri Slaby <jirislaby-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
		  Jiri Kosina <jkosina-AlSwsSmVLrQ@public.gmane.org>
		  Jiri Slaby <jirislaby-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
		  Denys Vlasenko <vda.linux-gM/Ye1E23mwN+BqQ9rBEUg@public.gmane.org>


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11906
Subject		: 2.6.28-rc2 seems to fail at powering down the monitor when it should
Submitter	: Gene Heskett <gene.heskett-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2008-10-30 6:39 (18 days old)
References	: http://marc.info/?l=linux-kernel&m=122534879721424&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11905
Subject		: lots of extra timer interrupts costing 2W
Submitter	: Theodore Ts'o <tytso-3s7WtUTddSA@public.gmane.org>
Date		: 2008-10-30 2:18 (18 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fb02fbc14d17837b4b7b02dbb36142c16a7bf208
References	: http://marc.info/?l=linux-kernel&m=122533314305315&w=4
		  http://marc.info/?l=linux-kernel&m=122541849114444&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11858
Subject		: Timeout regression introduced by 242f9dcb8ba6f68fcd217a119a7648a4f69290e9
Submitter	: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
Date		: 2008-10-26 9:46 (22 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=242f9dcb8ba6f68fcd217a119a7648a4f69290e9
References	: http://marc.info/?l=linux-kernel&m=122501447326698&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11849
Subject		: default IRQ affinity change in v2.6.27 (breaking several SMP PPC based systems)
Submitter	: Kumar Gala <galak-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org>
Date		: 2008-10-24 12:45 (24 days old)
References	: http://marc.info/?l=linux-kernel&m=122485245924125&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11822
Subject		: ACPI Warning (nspredef-0858): _SB_.PCI0.LPC_.EC__.BAT0._BIF: Return Package type mismatch at index 9 - found Buffer, expected String [20080926]
Submitter	: Len Brown <len.brown-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2008-10-25 01:26 (23 days old)
Handled-By	: Robert Moore <Robert.Moore-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>


Regressions with patches
------------------------

Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12047
Subject		: ACPI toshiba: only register rfkill if bt is enabled
Submitter	: Andrey Borzenkov <arvidjaar-JGs/UdohzUI@public.gmane.org>
Date		: 2008-10-28 19:10 (20 days old)
References	: http://marc.info/?l=linux-kernel&m=122522113619025&w=2
Handled-By	: Frederik Deweerdt <frederik.deweerdt-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122526843117478&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12042
Subject		: USB hid blocks USB port in 2.6.28rc3
Submitter	: Andi Kleen <andi-Vw/NltI1exuRpAAqCnN02g@public.gmane.org>
Date		: 2008-11-11 0:31 (6 days old)
References	: http://marc.info/?l=linux-usb&m=122636246522848&w=4
Handled-By	: Alan Stern <stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz@public.gmane.org>
		  Jiri Slaby <jirislaby-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122661425620251&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12038
Subject		: Suspend regression in stable kernel 2.6.27.4 on Mac mini Core Duo
Submitter	: Tino Keitel <tino.keitel-Mmb7MZpHnFY@public.gmane.org>
Date		: 2008-11-09 20:28 (8 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=66036f5862883fcc9f7ff8550685a5a3de1a57e4
References	: http://marc.info/?l=linux-kernel&m=122626258429689&w=4
Handled-By	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122661443120581&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12020
Subject		: scsi_times_out NULL pointer dereference
Submitter	: Bernd Schubert <bs-PKu+Ek1N2UGzQB+pC5nmwQ@public.gmane.org>
Date		: 2008-11-13 10:30 (4 days old)
Handled-By	: James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>
Patch		: http://bugzilla.kernel.org/show_bug.cgi?id=12020#c4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11989
Subject		: Suspend failure on NForce4-based boards due to chanes in stop_machine
Submitter	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
Date		: 2008-11-03 0:28 (14 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc
References	: http://marc.info/?l=linux-kernel&m=122567187604356&w=4
Handled-By	: Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org>
Patch		: http://lkml.org/lkml/2008/11/15/69


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11982
Subject		: Fan level 7 after resume wit 2.6.28-rc3
Submitter	: Tino Keitel <tino.keitel-rAwCM5oiXHA@public.gmane.org>
Date		: 2008-11-05 7:33 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=122587043409186&w=4
Handled-By	: Henrique de Moraes Holschuh <hmh-N3TV7GIv+o9fyO9Q7EP/yw@public.gmane.org>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=18744&action=view


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11954
Subject		: sysprof tracer ABI regression in 2.6.28rc1
Submitter	: Eric Anholt <eric-WhKQ6XTQaPysTnJN9+BGXg@public.gmane.org>
Date		: 2008-11-04 22:14 (13 days old)
Handled-By	: Eric Anholt <eric-WhKQ6XTQaPysTnJN9+BGXg@public.gmane.org>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=18683&action=view


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11925
Subject		: cdrom: missing compat ioctls
Submitter	: Andreas Schwab <schwab-l3A5Bk7waGM@public.gmane.org>
Date		: 2008-10-31 14:02 (17 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=33c2dca4957bd0da3e1af7b96d0758d97e708ef6
Handled-By	: Andreas Schwab <schwab-l3A5Bk7waGM@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122548923531545&w=2


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11911
Subject		: new PCMCIA device instance after resume - orinoco can't download firmware
Submitter	: Andrey Borzenkov <arvidjaar-JGs/UdohzUI@public.gmane.org>
Date		: 2008-10-28 19:19 (20 days old)
References	: http://marc.info/?l=linux-wireless&m=122522165719760&w=4
Handled-By	: Dave <kilroyd-gM/Ye1E23mwN+BqQ9rBEUg@public.gmane.org>
Patch		: http://marc.info/?l=linux-wireless&m=122539058601588&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11903
Subject		: regression: vmalloc easily fail
Submitter	: Glauber Costa <glommer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Date		: 2008-10-28 20:59 (20 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=db64fe02258f1507e13fe5212a989922323685ce
References	: http://marc.info/?l=linux-kernel&m=122522755530998&w=4
Handled-By	: Glauber Costa <glommer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
		  Nick Piggin <npiggin-l3A5Bk7waGM@public.gmane.org>
		  Glauber Costa <glommer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122609055221549&w=4


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11899
Subject		: sometime boot failed on T61 laptop
Submitter	: alexs <alex.shi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2008-10-30 02:04 (18 days old)
References	: http://marc.info/?l=linux-kernel&m=122663989015147&w=4
Handled-By	: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
		  Yanmin Zhang <yanmin_zhang-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=18860&action=view


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11898
Subject		: mke2fs hang on AIC79 device.
Submitter	: alexs <alex.shi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2008-10-30 01:17 (18 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=f0c0a376d0fcd4c5579ecf5e95f88387cba85211
Handled-By	: James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>
		  Mike Christie <michaelc-hcNo3dDEHLuVc3sceRu5cw@public.gmane.org>
Patch		: http://bugzilla.kernel.org/show_bug.cgi?id=11898#c28
		  http://bugzilla.kernel.org/show_bug.cgi?id=11898#c36


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11875
Subject		: radeonfb lockup in .28-rc (bisected)
Submitter	: James Cloos <cloos-GRsvFm/Gh/pBDgjK7y7TUQ@public.gmane.org>
Date		: 2008-10-28 0:00 (20 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b1ee26bab14886350ba12a5c10cbc0696ac679bf
References	: http://marc.info/?l=linux-kernel&m=122515210200530&w=4
		  http://lkml.org/lkml/2008/11/10/12
Handled-By	: Benjamin Herrenschmidt <benh-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org>
Patch		: http://lkml.org/lkml/2008/11/10/12


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11828
Subject		: Linux 2.6.27-git3: no SD card reader
Submitter	: J.A. Magallón <jamagallon-sh/6fXdz2Rs@public.gmane.org>
Date		: 2008-10-14 0:54 (34 days old)
References	: http://marc.info/?l=linux-kernel&m=122394573904699&w=4
Handled-By	: Pierre Ossman <drzeus-list-p3sGCRWkH8CeZLLa646FqQ@public.gmane.org>
Patch		: http://marc.info/?l=linux-pci&m=122662042229570&w=2


For details, please visit the bug entries and follow the links given in
references.

As you can see, there is a Bugzilla entry for each of the listed regressions.
There also is a Bugzilla entry used for tracking the regressions from 2.6.27,
unresolved as well as resolved, at:

http://bugzilla.kernel.org/show_bug.cgi?id=11808

Please let me know if there are any Bugzilla entries that should be added to
the list in there.

Thanks,
Rafael

[Bug #11822] ACPI Warning (nspredef-0858): _SB_.PCI0.LPC_.EC__.BAT0._BIF: Return Package type mismatch at index 9 - found Buffer, expected String [20080926]

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11822
Subject		: ACPI Warning (nspredef-0858): _SB_.PCI0.LPC_.EC__.BAT0._BIF: Return Package type mismatch at index 9 - found Buffer, expected String [20080926]
Submitter	: Len Brown <len.brown-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2008-10-25 01:26 (23 days old)
Handled-By	: Robert Moore <Robert.Moore-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

[Bug #11849] default IRQ affinity change in v2.6.27 (breaking several SMP PPC based systems)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11849
Subject		: default IRQ affinity change in v2.6.27 (breaking several SMP PPC based systems)
Submitter	: Kumar Gala <galak-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org>
Date		: 2008-10-24 12:45 (24 days old)
References	: http://marc.info/?l=linux-kernel&m=122485245924125&w=4

[Bug #11858] Timeout regression introduced by 242f9dcb8ba6f68fcd217a119a7648a4f69290e9

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11858
Subject		: Timeout regression introduced by 242f9dcb8ba6f68fcd217a119a7648a4f69290e9
Submitter	: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
Date		: 2008-10-26 9:46 (22 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=242f9dcb8ba6f68fcd217a119a7648a4f69290e9
References	: http://marc.info/?l=linux-kernel&m=122501447326698&w=4

[Bug #11875] radeonfb lockup in .28-rc (bisected)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11875
Subject		: radeonfb lockup in .28-rc (bisected)
Submitter	: James Cloos <cloos-GRsvFm/Gh/pBDgjK7y7TUQ@public.gmane.org>
Date		: 2008-10-28 0:00 (20 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b1ee26bab14886350ba12a5c10cbc0696ac679bf
References	: http://marc.info/?l=linux-kernel&m=122515210200530&w=4
		  http://lkml.org/lkml/2008/11/10/12
Handled-By	: Benjamin Herrenschmidt <benh-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org>
Patch		: http://lkml.org/lkml/2008/11/10/12

[Bug #11903] regression: vmalloc easily fail

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11903
Subject		: regression: vmalloc easily fail
Submitter	: Glauber Costa <glommer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Date		: 2008-10-28 20:59 (20 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=db64fe02258f1507e13fe5212a989922323685ce
References	: http://marc.info/?l=linux-kernel&m=122522755530998&w=4
Handled-By	: Glauber Costa <glommer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
		  Nick Piggin <npiggin-l3A5Bk7waGM@public.gmane.org>
		  Glauber Costa <glommer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122609055221549&w=4

[Bug #11898] mke2fs hang on AIC79 device.

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11898
Subject		: mke2fs hang on AIC79 device.
Submitter	: alexs <alex.shi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2008-10-30 01:17 (18 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=f0c0a376d0fcd4c5579ecf5e95f88387cba85211
Handled-By	: James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>
		  Mike Christie <michaelc-hcNo3dDEHLuVc3sceRu5cw@public.gmane.org>
Patch		: http://bugzilla.kernel.org/show_bug.cgi?id=11898#c28
		  http://bugzilla.kernel.org/show_bug.cgi?id=11898#c36

[Bug #11899] sometime boot failed on T61 laptop

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11899
Subject		: sometime boot failed on T61 laptop
Submitter	: alexs <alex.shi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2008-10-30 02:04 (18 days old)
References	: http://marc.info/?l=linux-kernel&m=122663989015147&w=4
Handled-By	: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
		  Yanmin Zhang <yanmin_zhang-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=18860&action=view

[Bug #11905] lots of extra timer interrupts costing 2W

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11905
Subject		: lots of extra timer interrupts costing 2W
Submitter	: Theodore Ts'o <tytso-3s7WtUTddSA@public.gmane.org>
Date		: 2008-10-30 2:18 (18 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fb02fbc14d17837b4b7b02dbb36142c16a7bf208
References	: http://marc.info/?l=linux-kernel&m=122533314305315&w=4
		  http://marc.info/?l=linux-kernel&m=122541849114444&w=4

[Bug #11911] new PCMCIA device instance after resume - orinoco can't download firmware

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11911
Subject		: new PCMCIA device instance after resume - orinoco can't download firmware
Submitter	: Andrey Borzenkov <arvidjaar-JGs/UdohzUI@public.gmane.org>
Date		: 2008-10-28 19:19 (20 days old)
References	: http://marc.info/?l=linux-wireless&m=122522165719760&w=4
Handled-By	: Dave <kilroyd-gM/Ye1E23mwN+BqQ9rBEUg@public.gmane.org>
Patch		: http://marc.info/?l=linux-wireless&m=122539058601588&w=4

[Bug #11906] 2.6.28-rc2 seems to fail at powering down the monitor when it should

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11906
Subject		: 2.6.28-rc2 seems to fail at powering down the monitor when it should
Submitter	: Gene Heskett <gene.heskett-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2008-10-30 6:39 (18 days old)
References	: http://marc.info/?l=linux-kernel&m=122534879721424&w=4

[Bug #11954] sysprof tracer ABI regression in 2.6.28rc1

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11954
Subject		: sysprof tracer ABI regression in 2.6.28rc1
Submitter	: Eric Anholt <eric-WhKQ6XTQaPysTnJN9+BGXg@public.gmane.org>
Date		: 2008-11-04 22:14 (13 days old)
Handled-By	: Eric Anholt <eric-WhKQ6XTQaPysTnJN9+BGXg@public.gmane.org>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=18683&action=view

[Bug #11913] USB/INPUT: slab error in cache_alloc_debugcheck_after(): double free?

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11913
Subject		: USB/INPUT: slab error in cache_alloc_debugcheck_after(): double free?
Submitter	: Helge Deller <deller-Mmb7MZpHnFY@public.gmane.org>
Date		: 2008-10-30 23:11 (18 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=cb8f488c33539f096580e202f5438a809195008f
References	: http://marc.info/?l=linux-kernel&m=122540833301394&w=4
Handled-By	: Jiri Kosina <jkosina-AlSwsSmVLrQ@public.gmane.org>
		  Jiri Slaby <jirislaby-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
		  Jiri Kosina <jkosina-AlSwsSmVLrQ@public.gmane.org>
		  Jiri Slaby <jirislaby-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
		  Denys Vlasenko <vda.linux-gM/Ye1E23mwN+BqQ9rBEUg@public.gmane.org>

[Bug #11925] cdrom: missing compat ioctls

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11925
Subject		: cdrom: missing compat ioctls
Submitter	: Andreas Schwab <schwab-l3A5Bk7waGM@public.gmane.org>
Date		: 2008-10-31 14:02 (17 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=33c2dca4957bd0da3e1af7b96d0758d97e708ef6
Handled-By	: Andreas Schwab <schwab-l3A5Bk7waGM@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122548923531545&w=2

[Bug #11947] 2.6.28-rc VC switching with Intel graphics broken

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11947
Subject		: 2.6.28-rc VC switching with Intel graphics broken
Submitter	: Romano Giannetti <romano.giannetti-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2008-11-03 12:10 (14 days old)
Handled-By	: Jesse Barnes <jbarnes-Y1mF5jBUw70BENJcbMCuUQ@public.gmane.org>

[Bug #11965] regression introduced by - timers: fix itimer/many thread hang

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11965
Subject		: regression introduced by - timers: fix itimer/many thread hang
Submitter	: Doug Chapman <doug.chapman-VXdhtT5mjnY@public.gmane.org>
Date		: 2008-11-06 11:03 (11 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=f06febc96ba8e0af80bcc3eaec0a109e88275fac
References	: http://marc.info/?l=linux-kernel&m=122596943416648&w=4
Handled-By	: Frank Mayhar <fmayhar-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
		  Peter Zijlstra <peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
		  Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>

[Bug #11958] [2.6.27.x => 2.6.28-rc3] Xorg crash with xf86MapVidMem error

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11958
Subject		: [2.6.27.x => 2.6.28-rc3] Xorg crash with xf86MapVidMem error
Submitter	: Tomasz Chmielewski <tch-Nem3ZqsbT/g@public.gmane.org>
Date		: 2008-11-05 05:37 (12 days old)

[Bug #11970] gettimeofday return a old time in mmbench

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11970
Subject		: gettimeofday return a old time in mmbench
Submitter	: alexs <alex.shi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Date		: 2008-11-06 23:57 (11 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=99ebcf8285df28f32fd2d1c19a7166e70f00309c
Handled-By	: Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org>
		  Thomas Gleixner <tglx-hfZtesqFncYOwBW4kG4KsQ@public.gmane.org>

[Bug #11982] Fan level 7 after resume wit 2.6.28-rc3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11982
Subject		: Fan level 7 after resume wit 2.6.28-rc3
Submitter	: Tino Keitel <tino.keitel-rAwCM5oiXHA@public.gmane.org>
Date		: 2008-11-05 7:33 (12 days old)
References	: http://marc.info/?l=linux-kernel&m=122587043409186&w=4
Handled-By	: Henrique de Moraes Holschuh <hmh-N3TV7GIv+o9fyO9Q7EP/yw@public.gmane.org>
Patch		: http://bugzilla.kernel.org/attachment.cgi?id=18744&action=view

[Bug #11996] Tracing framework regression in 2.6.28-rc3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11996
Subject		: Tracing framework regression in 2.6.28-rc3
Submitter	: Pekka Paalanen <pq-X3B1VOXEql0@public.gmane.org>
Date		: 2008-11-09 10:13 (8 days old)
References	: http://marc.info/?l=linux-kernel&m=122624392229317&w=4
Handled-By	: Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org>

[Bug #11989] Suspend failure on NForce4-based boards due to chanes in stop_machine

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11989
Subject		: Suspend failure on NForce4-based boards due to chanes in stop_machine
Submitter	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
Date		: 2008-11-03 0:28 (14 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c9583e55fa2b08a230c549bd1e3c0bde6c50d9cc
References	: http://marc.info/?l=linux-kernel&m=122567187604356&w=4
Handled-By	: Rusty Russell <rusty-8n+1lVoiYb80n/F98K4Iww@public.gmane.org>
Patch		: http://lkml.org/lkml/2008/11/15/69

[Bug #12019] Major increase in the number of wakeups from C3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12019
Subject		: Major increase in the number of wakeups from C3
Submitter	: Mircea Gherzan <mgherzan-0aWM6n0cdD1zL9IiBdi9GK9KmVUGpFAF@public.gmane.org>
Date		: 2008-11-13 07:02 (4 days old)

[Bug #12020] scsi_times_out NULL pointer dereference

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12020
Subject		: scsi_times_out NULL pointer dereference
Submitter	: Bernd Schubert <bs-PKu+Ek1N2UGzQB+pC5nmwQ@public.gmane.org>
Date		: 2008-11-13 10:30 (4 days old)
Handled-By	: James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>
Patch		: http://bugzilla.kernel.org/show_bug.cgi?id=12020#c4

[Bug #12040] iwlagn driver segfault in 2.6.28-rc3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12040
Subject		: iwlagn driver segfault in 2.6.28-rc3
Submitter	: Lukas Hejtmanek <xhejtman-8qz54MUs51PtwjQa/ONI9g@public.gmane.org>
Date		: 2008-11-11 14:34 (6 days old)
References	: http://marc.info/?l=linux-kernel&m=122641417331593&w=4
Handled-By	: reinette chatre <reinette.chatre-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

[Bug #12028] i915 DRM is broken in 2.6.28-rc4

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12028
Subject		: i915 DRM is broken in 2.6.28-rc4
Submitter	: Adam Tkac <vonsch-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Date		: 2008-11-14 01:50 (3 days old)

[Bug #12038] Suspend regression in stable kernel 2.6.27.4 on Mac mini Core Duo

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12038
Subject		: Suspend regression in stable kernel 2.6.27.4 on Mac mini Core Duo
Submitter	: Tino Keitel <tino.keitel-Mmb7MZpHnFY@public.gmane.org>
Date		: 2008-11-09 20:28 (8 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=66036f5862883fcc9f7ff8550685a5a3de1a57e4
References	: http://marc.info/?l=linux-kernel&m=122626258429689&w=4
Handled-By	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122661443120581&w=4

[Bug #12034] snd-hda-intel on Realtek ALC268 chip shows only Master volume (for playback)

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12034
Subject		: snd-hda-intel on Realtek ALC268 chip shows only Master volume (for playback)
Submitter	: Sergey <azure-IGUVrPOATOfsG83rWm+8vg@public.gmane.org>
Date		: 2008-11-15 04:20 (2 days old)

[Bug #12031] DRM enabled kernel hangs hard on resume on x60

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12031
Subject		: DRM enabled kernel hangs hard on resume on x60
Submitter	: Jens Axboe <jens.axboe-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Date		: 2008-11-12 18:42 (5 days old)
References	: http://marc.info/?l=linux-kernel&m=122651551216820&w=4
Handled-By	: Jesse Barnes <jbarnes-Y1mF5jBUw70BENJcbMCuUQ@public.gmane.org>

[Bug #12047] ACPI toshiba: only register rfkill if bt is enabled

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12047
Subject		: ACPI toshiba: only register rfkill if bt is enabled
Submitter	: Andrey Borzenkov <arvidjaar-JGs/UdohzUI@public.gmane.org>
Date		: 2008-10-28 19:10 (20 days old)
References	: http://marc.info/?l=linux-kernel&m=122522113619025&w=2
Handled-By	: Frederik Deweerdt <frederik.deweerdt-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122526843117478&w=2

[Bug #12049] Oops in acpi_system_wakeup_device_seq_show

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12049
Subject		: Oops in acpi_system_wakeup_device_seq_show
Submitter	: Bruno Prémont <bonbons-ud5FBsm0p/xEiooADzr8i9i2O/JbrIOy@public.gmane.org>
Date		: 2008-11-16 13:04 (1 days old)
First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0794469da3f7b2093575cbdfc1108308dd3641ce
References	: http://marc.info/?l=linux-kernel&m=122684070215520&w=4

[Bug #12042] USB hid blocks USB port in 2.6.28rc3

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12042
Subject		: USB hid blocks USB port in 2.6.28rc3
Submitter	: Andi Kleen <andi-Vw/NltI1exuRpAAqCnN02g@public.gmane.org>
Date		: 2008-11-11 0:31 (6 days old)
References	: http://marc.info/?l=linux-usb&m=122636246522848&w=4
Handled-By	: Alan Stern <stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz@public.gmane.org>
		  Jiri Slaby <jirislaby-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Patch		: http://marc.info/?l=linux-kernel&m=122661425620251&w=2

[Bug #12041] 2.6.28-rc4 mem_cgroup_charge_common panic

[Rafael J. Wysocki]

This message has been generated automatically as a part of a report
of recent regressions.

The following bug entry is on the current list of known regressions
from 2.6.27.  Please verify if it still should be listed and let me know
(either way).


Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12041
Subject		: 2.6.28-rc4 mem_cgroup_charge_common panic
Submitter	: Badari Pulavarty <pbadari-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
Date		: 2008-11-10 21:43 (7 days old)
References	: http://marc.info/?l=linux-kernel&m=122635343008147&w=4
Handled-By	: KAMEZAWA Hiroyuki <kamezawa.hiroyu-+CUm20s59erQFUHtdCDX3A@public.gmane.org>

Re: 2.6.28-rc5: Reported regressions from 2.6.27

[Alan Cox]

>      the majority of these patches have not been merged for over two weeks
>      (for almost three weeks in many cases and for over a _month_ in one case).
>      IMO, this is insane.]

Some of them are quite serious too - some stuff is basically unusable in
2.6.28rc due to the vmalloc bug.

Is there any reason why someone (Rafael ?) shouldn't simply submit all of
those patches that look sensible, are reported to fix regressions and
whose maintainer has not provided a reason to NOT apply them into the
tree ?

Alan

Re: [Bug #11911] new PCMCIA device instance after resume - orinoco can't download firmware

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]

On Sunday 16 November 2008, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.27.  Please verify if it still should be listed and let me know
> (either way).
> 

Still valid in rc5. As indicated by

http://marc.info/?l=linux-kernel&m=122679210409961&w=2

pull request was apparently lost.

> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11911
> Subject		: new PCMCIA device instance after resume - orinoco can't download firmware
> Submitter	: Andrey Borzenkov <arvidjaar-JGs/UdohzUI@public.gmane.org>
> Date		: 2008-10-28 19:19 (20 days old)
> References	: http://marc.info/?l=linux-wireless&m=122522165719760&w=4
> Handled-By	: Dave <kilroyd-gM/Ye1E23mwN+BqQ9rBEUg@public.gmane.org>
> Patch		: http://marc.info/?l=linux-wireless&m=122539058601588&w=4
> 
> 
> 



[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: 2.6.28-rc5: Reported regressions from 2.6.27

[Linus Torvalds]

On Sun, 16 Nov 2008, Rafael J. Wysocki wrote:
>
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12049
> Subject		: Oops in acpi_system_wakeup_device_seq_show
> Submitter	: Bruno Prémont <bonbons-ud5FBsm0p/xEiooADzr8i9i2O/JbrIOy@public.gmane.org>
> Date		: 2008-11-16 13:04 (1 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0794469da3f7b2093575cbdfc1108308dd3641ce
> References	: http://marc.info/?l=linux-kernel&m=122684070215520&w=4

This already got fixed: commit 77fb61a04a0483ad274ce5c51b02c46c12db3693.

> Regressions with patches
> ------------------------
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12042
> Subject		: USB hid blocks USB port in 2.6.28rc3
> Submitter	: Andi Kleen <andi-Vw/NltI1exuRpAAqCnN02g@public.gmane.org>
> Date		: 2008-11-11 0:31 (6 days old)
> References	: http://marc.info/?l=linux-usb&m=122636246522848&w=4
> Handled-By	: Alan Stern <stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz@public.gmane.org>
> 		  Jiri Slaby <jirislaby-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> Patch		: http://marc.info/?l=linux-kernel&m=122661425620251&w=2

commit 131d3a7a009d56a96cc7117b4e9d0c90c2e2a1dc

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11954
> Subject		: sysprof tracer ABI regression in 2.6.28rc1
> Submitter	: Eric Anholt <eric-WhKQ6XTQaPysTnJN9+BGXg@public.gmane.org>
> Date		: 2008-11-04 22:14 (13 days old)
> Handled-By	: Eric Anholt <eric-WhKQ6XTQaPysTnJN9+BGXg@public.gmane.org>
> Patch		: http://bugzilla.kernel.org/attachment.cgi?id=18683&action=view

commit 072ba49838b42c873c496d72c91bb237914cf3b6

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11911
> Subject		: new PCMCIA device instance after resume - orinoco can't download firmware
> Submitter	: Andrey Borzenkov <arvidjaar-JGs/UdohzUI@public.gmane.org>
> Date		: 2008-10-28 19:19 (20 days old)
> References	: http://marc.info/?l=linux-wireless&m=122522165719760&w=4
> Handled-By	: Dave <kilroyd-gM/Ye1E23mwN+BqQ9rBEUg@public.gmane.org>
> Patch		: http://marc.info/?l=linux-wireless&m=122539058601588&w=4

commit e689597fe890cf22e23195037aa668c39b25ae4b

			Linus

Re: 2.6.28-rc5: Reported regressions from 2.6.27

[Rafael J. Wysocki]

On Sunday, 16 of November 2008, Linus Torvalds wrote:
> 
> On Sun, 16 Nov 2008, Rafael J. Wysocki wrote:
> >
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12049
> > Subject		: Oops in acpi_system_wakeup_device_seq_show
> > Submitter	: Bruno Prémont <bonbons@linux-vserver.org>
> > Date		: 2008-11-16 13:04 (1 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0794469da3f7b2093575cbdfc1108308dd3641ce
> > References	: http://marc.info/?l=linux-kernel&m=122684070215520&w=4
> 
> This already got fixed: commit 77fb61a04a0483ad274ce5c51b02c46c12db3693.
> 
> > Regressions with patches
> > ------------------------
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12042
> > Subject		: USB hid blocks USB port in 2.6.28rc3
> > Submitter	: Andi Kleen <andi@firstfloor.org>
> > Date		: 2008-11-11 0:31 (6 days old)
> > References	: http://marc.info/?l=linux-usb&m=122636246522848&w=4
> > Handled-By	: Alan Stern <stern@rowland.harvard.edu>
> > 		  Jiri Slaby <jirislaby@gmail.com>
> > Patch		: http://marc.info/?l=linux-kernel&m=122661425620251&w=2
> 
> commit 131d3a7a009d56a96cc7117b4e9d0c90c2e2a1dc
> 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11954
> > Subject		: sysprof tracer ABI regression in 2.6.28rc1
> > Submitter	: Eric Anholt <eric@anholt.net>
> > Date		: 2008-11-04 22:14 (13 days old)
> > Handled-By	: Eric Anholt <eric@anholt.net>
> > Patch		: http://bugzilla.kernel.org/attachment.cgi?id=18683&action=view
> 
> commit 072ba49838b42c873c496d72c91bb237914cf3b6
> 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11911
> > Subject		: new PCMCIA device instance after resume - orinoco can't download firmware
> > Submitter	: Andrey Borzenkov <arvidjaar@mail.ru>
> > Date		: 2008-10-28 19:19 (20 days old)
> > References	: http://marc.info/?l=linux-wireless&m=122522165719760&w=4
> > Handled-By	: Dave <kilroyd@googlemail.com>
> > Patch		: http://marc.info/?l=linux-wireless&m=122539058601588&w=4
> 
> commit e689597fe890cf22e23195037aa668c39b25ae4b

Thanks a lot, closed all of them.

Best,
Rafael
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.28-rc5: Reported regressions from 2.6.27

[Maxim Levitsky]

Just to note, here on acer aspire one suspend to disk is broken too.
System suspends, and then hangs.

If I shutdown the system, and then attempt resume, it resumes, and then hangs.

Best regards,
	Maxim Levitsky

Re: [Bug #11996] Tracing framework regression in 2.6.28-rc3

[Pekka Paalanen]

On Sun, 16 Nov 2008 17:35:18 +0100 (CET)
"Rafael J. Wysocki" <rjw-KKrjLPT3xs0@public.gmane.org> wrote:

> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.27.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11996
> Subject		: Tracing framework regression in 2.6.28-rc3
> Submitter	: Pekka Paalanen <pq-X3B1VOXEql0@public.gmane.org>
> Date		: 2008-11-09 10:13 (8 days old)
> References	: http://marc.info/?l=linux-kernel&m=122624392229317&w=4
> Handled-By	: Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org>

Steve, Ingo, did you get into an agreement on the patch?
What should I test?

I see -rc5 is out, but I didn't spot the fix in the changelog.

(The ring buffer NULL dereference on resize / unallocated max tracer.)

-- 
Pekka Paalanen
http://www.iki.fi/pq/

Re: [Bug #11996] Tracing framework regression in 2.6.28-rc3

[Steven Rostedt]

On Mon, 17 Nov 2008, Pekka Paalanen wrote:

> On Sun, 16 Nov 2008 17:35:18 +0100 (CET)
> "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> 
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.27.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11996
> > Subject		: Tracing framework regression in 2.6.28-rc3
> > Submitter	: Pekka Paalanen <pq@iki.fi>
> > Date		: 2008-11-09 10:13 (8 days old)
> > References	: http://marc.info/?l=linux-kernel&m=122624392229317&w=4
> > Handled-By	: Steven Rostedt <rostedt@goodmis.org>
> 
> Steve, Ingo, did you get into an agreement on the patch?
> What should I test?
> 
> I see -rc5 is out, but I didn't spot the fix in the changelog.
> 
> (The ring buffer NULL dereference on resize / unallocated max tracer.)

Ingo's solution was to have the ring_buffer_resize return success on NULL 
buffer being passed in. Although I agree that it should not crash when 
passed a NULL pointer, I feel that a NULL pointer should return a -1 
(failure). The caller of the code (one place in kernel/trace/trace.c) 
could simply check if the buffer was allocated, and if not, simply ignore 
it.

I agree with Ingo that my original solution was too much churn. But the 
simple if statement and "indent" change is what I feel to be the solution, 
not letting the ring buffer return success on NULL pointer.

-- Steve

Re: [Bug #11996] Tracing framework regression in 2.6.28-rc3

[Ingo Molnar]

* Pekka Paalanen <pq-X3B1VOXEql0@public.gmane.org> wrote:

> On Sun, 16 Nov 2008 17:35:18 +0100 (CET)
> "Rafael J. Wysocki" <rjw-KKrjLPT3xs0@public.gmane.org> wrote:
> 
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.27.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11996
> > Subject		: Tracing framework regression in 2.6.28-rc3
> > Submitter	: Pekka Paalanen <pq-X3B1VOXEql0@public.gmane.org>
> > Date		: 2008-11-09 10:13 (8 days old)
> > References	: http://marc.info/?l=linux-kernel&m=122624392229317&w=4
> > Handled-By	: Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org>
> 
> Steve, Ingo, did you get into an agreement on the patch? What should 
> I test?

the fix is queued up in tip/tracing/urgent, will send it to Linus 
today or tomorrow.

	Ingo

Re: 2.6.28-rc5: Reported regressions from 2.6.27

[James Bottomley]

On Sun, 2008-11-16 at 18:29 +0000, Alan Cox wrote:
> >      the majority of these patches have not been merged for over two weeks
> >      (for almost three weeks in many cases and for over a _month_ in one case).
> >      IMO, this is insane.]
> 
> Some of them are quite serious too - some stuff is basically unusable in
> 2.6.28rc due to the vmalloc bug.
> 
> Is there any reason why someone (Rafael ?) shouldn't simply submit all of
> those patches that look sensible, are reported to fix regressions and
> whose maintainer has not provided a reason to NOT apply them into the
> tree ?

There is for SCSI.  Our two bugzilla entries each have several patches
(one has two, the other has four).  The patches listed in the
regressions aren't necessarily going to be the ones applied (depending
on how the arguing and testing goes).

James

Re: [Bug #11954] sysprof tracer ABI regression in 2.6.28rc1

[Soeren Sandmann]

"Rafael J. Wysocki" <rjw-KKrjLPT3xs0@public.gmane.org> writes:

> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.27.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11954
> Subject		: sysprof tracer ABI regression in 2.6.28rc1
> Submitter	: Eric Anholt <eric-WhKQ6XTQaPysTnJN9+BGXg@public.gmane.org>
> Date		: 2008-11-04 22:14 (13 days old)
> Handled-By	: Eric Anholt <eric-WhKQ6XTQaPysTnJN9+BGXg@public.gmane.org>
> Patch		: http://bugzilla.kernel.org/attachment.cgi?id=18683&action=view

There are no ABI guarantees for the sysprof tracer at this point,
given that it is in debugfs and that there is no working userspace
code using it.



Soren

Re: [Bug #11954] sysprof tracer ABI regression in 2.6.28rc1

[Ingo Molnar]

* Soeren Sandmann <sandmann-SSQ8kajmRZpknbxzx/v8hQ@public.gmane.org> wrote:

> There are no ABI guarantees for the sysprof tracer at this point, 
> given that it is in debugfs [...]

to avoid situations like this, it would be nice to print out the 
format string itself in the trace header, in printf format string 
style. That way if there's a trivial expansion in trace format, it can 
be detected (and even followed) by user-space - without some ugly 
version based API.

	Ingo

Re: [Bug #11947] 2.6.28-rc VC switching with Intel graphics broken

[Romano Giannetti (lists)]

Rafael J. Wysocki wrote:

> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=11947
> Subject		: 2.6.28-rc VC switching with Intel graphics broken

Continue to malfunction. More details on the bugzilla entry. But basically, I had:

- one complete lock on VC switching (not switching back, just pressing ctrl-alt-f1).
- 3 two-plus minutes delay on VC switching
- a complete freeze on resuming from ram (not even SysRq-b working).

OTOH, it seems that VC malfunctioning happens only with "visual effect" aka 
compiz on.

Back to 2.6.27.5. This is quite worrying now... should I try to start a
bisection? Although I know from experience that bisecting over a -rc1 is quite
painful.

Romano


-- 
Sorry for the disclaimer --- ¡I cannot stop it!


--
La presente comunicación tiene carácter confidencial y es para el exclusivo uso del destinatario indicado en la misma. Si Ud. no es el destinatario indicado, le informamos que cualquier forma de distribución, reproducción o uso de esta comunicación y/o de la información contenida en la misma están estrictamente prohibidos por la ley. Si Ud. ha recibido esta comunicación por error, por favor, notifíquelo inmediatamente al remitente contestando a este mensaje y proceda a continuación a destruirlo. Gracias por su colaboración.

This communication contains confidential information. It is for the exclusive use of the intended addressee. If you are not the intended addressee, please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited by law. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this message. Thank you for your cooperation. 

Re: [Bug #11982] Fan level 7 after resume wit 2.6.28-rc3

[Tino Keitel]

On Sun, Nov 16, 2008 at 17:35:18 +0100, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.27.  Please verify if it still should be listed and let me know
> (either way).

The fix seems to be still missing from HEAD, but I tested it and it
fixed the regression that I noticed.

Regards,
Tino

Re: 2.6.28-rc5: Reported regressions from 2.6.27

[Rafael J. Wysocki]

On Sunday, 16 of November 2008, Maxim Levitsky wrote:
> Just to note, here on acer aspire one suspend to disk is broken too.
> System suspends, and then hangs.

At which point exactly does it hang?

What's the most recent kernel you tried?

> If I shutdown the system, and then attempt resume, it resumes, and then hangs.

Again, at which point does it hang exactly?

I assume this doesn't happen with 2.6.27.  Is this correct?

Rafael

Re: [Bug #12038] Suspend regression in stable kernel 2.6.27.4 on Mac mini Core Duo

[Tino Keitel]

On Sun, Nov 16, 2008 at 17:35:21 +0100, Rafael J. Wysocki wrote:
> This message has been generated automatically as a part of a report
> of recent regressions.
> 
> The following bug entry is on the current list of known regressions
> from 2.6.27.  Please verify if it still should be listed and let me know
> (either way).
> 
> 
> Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12038
> Subject		: Suspend regression in stable kernel 2.6.27.4 on Mac mini Core Duo
> Submitter	: Tino Keitel <tino.keitel@gmx.de>
> Date		: 2008-11-09 20:28 (8 days old)
> First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=66036f5862883fcc9f7ff8550685a5a3de1a57e4
> References	: http://marc.info/?l=linux-kernel&m=122626258429689&w=4
> Handled-By	: Rafael J. Wysocki <rjw@sisk.pl>
> Patch		: http://marc.info/?l=linux-kernel&m=122661443120581&w=4

The patch works fine here. I applied it to 2.6.27.5 and resume works.

Regards,
Tino

Re: [Bug #12038] Suspend regression in stable kernel 2.6.27.4 on Mac mini Core Duo

[Rafael J. Wysocki]

On Monday, 17 of November 2008, Tino Keitel wrote:
> On Sun, Nov 16, 2008 at 17:35:21 +0100, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.27.  Please verify if it still should be listed and let me know
> > (either way).
> > 
> > 
> > Bug-Entry	: http://bugzilla.kernel.org/show_bug.cgi?id=12038
> > Subject		: Suspend regression in stable kernel 2.6.27.4 on Mac mini Core Duo
> > Submitter	: Tino Keitel <tino.keitel-Mmb7MZpHnFY@public.gmane.org>
> > Date		: 2008-11-09 20:28 (8 days old)
> > First-Bad-Commit: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=66036f5862883fcc9f7ff8550685a5a3de1a57e4
> > References	: http://marc.info/?l=linux-kernel&m=122626258429689&w=4
> > Handled-By	: Rafael J. Wysocki <rjw-KKrjLPT3xs0@public.gmane.org>
> > Patch		: http://marc.info/?l=linux-kernel&m=122661443120581&w=4
> 
> The patch works fine here. I applied it to 2.6.27.5 and resume works.

Thanks for testing, I'm going to push it upstream shortly.

Best,
Rafael

Re: [Bug #11954] sysprof tracer ABI regression in 2.6.28rc1

[Soeren Sandmann]

Ingo Molnar <mingo-X9Un+BFzKDI@public.gmane.org> writes:

> * Soeren Sandmann <sandmann-SSQ8kajmRZpknbxzx/v8hQ@public.gmane.org> wrote:
> 
> > There are no ABI guarantees for the sysprof tracer at this point, 
> > given that it is in debugfs [...]
> 
> to avoid situations like this, it would be nice to print out the 
> format string itself in the trace header, in printf format string 
> style. That way if there's a trivial expansion in trace format, it can 
> be detected (and even followed) by user-space - without some ugly 
> version based API.

It's not clear to me that sysprof userspace could realistically do
runtime detection. It would require (a) using sscanf() to parse the
output and (b) that only the types changed, not the actual content or
ordering. (a) is likely a non-starter performance-wise (the first
version actually did this and it was too slow), and (b) means there
are strict limits to what could be changed anyway.


Soren