patchwork.kernel.org down

patchwork.kernel.org down

[Sedat Dilek]

Hi,

just FYI: http://patchwork.kernel.org/ is down!

Regards,
- Sedat -

Re: patchwork.kernel.org down

[David Brown]

On Mon, Dec 20, 2010 at 09:12:58PM +0100, Sedat Dilek wrote:

> just FYI: http://patchwork.kernel.org/ is down!

git.kernel.org seems to be down as well.

David

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: patchwork.kernel.org down

[J.H.]

On 12/20/2010 12:49 PM, David Brown wrote:
> On Mon, Dec 20, 2010 at 09:12:58PM +0100, Sedat Dilek wrote:
> 
>> just FYI: http://patchwork.kernel.org/ is down!

More excitement from https://bugzilla.kernel.org/show_bug.cgi?id=20702

Anyone want to take a stab at it?  I know I'd be appreciative.  I am
running a debug kernel with everything I could find and enable I even
remotely thought might prove helpful.  Nothing has jumped out yet though.

> git.kernel.org seems to be down as well.

git isn't down, but it has been unbelievably busy.  Loads are *only* in
the 60's right now (they have been spiking above 300)

I did mention the loads and the spikes yesterday:

https://lkml.org/lkml/2010/12/20/317

and you can see the loads on the odins (the ones that deal with git) here:

http://cacti.kernel.org/graph_view.php?action=tree&tree_id=3

We are down from the 2700 or so simultaneous git processes to only about
1600 right now (which again the normal range is 100-200).  The onslaught
is subsiding, as it does with any big release, just no one knew it was
going to be nearly as big as it has turned out to be.

- John 'Warthog9' Hawley

Re: patchwork.kernel.org down

[Roland Dreier]

 > More excitement from https://bugzilla.kernel.org/show_bug.cgi?id=20702
 > 
 > Anyone want to take a stab at it?  I know I'd be appreciative.  I am
 > running a debug kernel with everything I could find and enable I even
 > remotely thought might prove helpful.  Nothing has jumped out yet though.

A little late, but perhaps running with the patch below might help us
make a bit of progress.  The idea is to dump the last /proc/net file
opened and closed, so we can at least have a clue as to where the crash
is coming from.  This should add lines like 

    last procfs open:  /proc/2443/net/arp
    last procfs close: /proc/2443/net/arp

to the oops output, so maybe we can zero in on things after you get a
crash with this applied.

 - R.

diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 6e8752c..c67b8d6 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -18,6 +18,7 @@
 
 #include <asm/stacktrace.h>
 
+void procfs_printk_last_file(void);
 
 int panic_on_unrecovered_nmi;
 int panic_on_io_nmi;
@@ -283,6 +284,7 @@ int __kprobes __die(const char *str, struct pt_regs *regs, long err)
 #endif
 	printk("\n");
 	sysfs_printk_last_file();
+	procfs_printk_last_file();
 	if (notify_die(DIE_OOPS, str, regs, err,
 			current->thread.trap_no, SIGSEGV) == NOTIFY_STOP)
 		return 1;
diff --git a/fs/proc/proc_net.c b/fs/proc/proc_net.c
index 9020ac1..1801cc1 100644
--- a/fs/proc/proc_net.c
+++ b/fs/proc/proc_net.c
@@ -26,6 +26,14 @@
 
 #include "internal.h"
 
+/* used in crash dumps to help with debugging */
+static char last_procfs_open[PATH_MAX];
+static char last_procfs_close[PATH_MAX];
+void procfs_printk_last_file(void)
+{
+	printk(KERN_EMERG "last procfs open:  %s\n", last_procfs_open);
+	printk(KERN_EMERG "last procfs close: %s\n", last_procfs_close);
+}
 
 static struct net *get_proc_net(const struct inode *inode)
 {
@@ -37,9 +45,14 @@ int seq_open_net(struct inode *ino, struct file *f,
 {
 	struct net *net;
 	struct seq_net_private *p;
+	char *n;
 
 	BUG_ON(size < sizeof(*p));
 
+	n = d_path(&f->f_path, last_procfs_open, sizeof(last_procfs_open));
+	if (!IS_ERR(n))
+		memmove(last_procfs_open, n, strlen(n) + 1);
+
 	net = get_proc_net(ino);
 	if (net == NULL)
 		return -ENXIO;
@@ -83,6 +96,11 @@ EXPORT_SYMBOL_GPL(single_open_net);
 int seq_release_net(struct inode *ino, struct file *f)
 {
 	struct seq_file *seq;
+	char *n;
+
+	n = d_path(&f->f_path, last_procfs_close, sizeof(last_procfs_close));
+	if (!IS_ERR(n))
+		memmove(last_procfs_close, n, strlen(n) + 1);
 
 	seq = f->private_data;
 

Re: patchwork.kernel.org down

[Randy Dunlap]

On Wed, 05 Jan 2011 14:52:17 -0800 Roland Dreier wrote:

>  > More excitement from https://bugzilla.kernel.org/show_bug.cgi?id=20702
>  > 
>  > Anyone want to take a stab at it?  I know I'd be appreciative.  I am
>  > running a debug kernel with everything I could find and enable I even
>  > remotely thought might prove helpful.  Nothing has jumped out yet though.
> 
> A little late, but perhaps running with the patch below might help us
> make a bit of progress.  The idea is to dump the last /proc/net file
> opened and closed, so we can at least have a clue as to where the crash
> is coming from.  This should add lines like 
> 
>     last procfs open:  /proc/2443/net/arp
>     last procfs close: /proc/2443/net/arp
> 
> to the oops output, so maybe we can zero in on things after you get a
> crash with this applied.
> 
>  - R.
> 
> diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
> index 6e8752c..c67b8d6 100644
> --- a/arch/x86/kernel/dumpstack.c
> +++ b/arch/x86/kernel/dumpstack.c
> @@ -18,6 +18,7 @@
>  
>  #include <asm/stacktrace.h>
>  
> +void procfs_printk_last_file(void);
>  
>  int panic_on_unrecovered_nmi;
>  int panic_on_io_nmi;
> @@ -283,6 +284,7 @@ int __kprobes __die(const char *str, struct pt_regs *regs, long err)
>  #endif
>  	printk("\n");
>  	sysfs_printk_last_file();
> +	procfs_printk_last_file();
>  	if (notify_die(DIE_OOPS, str, regs, err,
>  			current->thread.trap_no, SIGSEGV) == NOTIFY_STOP)
>  		return 1;
> diff --git a/fs/proc/proc_net.c b/fs/proc/proc_net.c
> index 9020ac1..1801cc1 100644
> --- a/fs/proc/proc_net.c
> +++ b/fs/proc/proc_net.c
> @@ -26,6 +26,14 @@
>  
>  #include "internal.h"
>  
> +/* used in crash dumps to help with debugging */
> +static char last_procfs_open[PATH_MAX];
> +static char last_procfs_close[PATH_MAX];
> +void procfs_printk_last_file(void)
> +{
> +	printk(KERN_EMERG "last procfs open:  %s\n", last_procfs_open);
> +	printk(KERN_EMERG "last procfs close: %s\n", last_procfs_close);

Let's not mislead the reader of a crash dump, please.  How about:

+	printk(KERN_EMERG "last /proc..net open:  %s\n", last_procfs_open);
+	printk(KERN_EMERG "last /proc..net close: %s\n", last_procfs_close);


> +}
>  
>  static struct net *get_proc_net(const struct inode *inode)
>  {
> @@ -37,9 +45,14 @@ int seq_open_net(struct inode *ino, struct file *f,
>  {
>  	struct net *net;
>  	struct seq_net_private *p;
> +	char *n;
>  
>  	BUG_ON(size < sizeof(*p));
>  
> +	n = d_path(&f->f_path, last_procfs_open, sizeof(last_procfs_open));
> +	if (!IS_ERR(n))
> +		memmove(last_procfs_open, n, strlen(n) + 1);
> +
>  	net = get_proc_net(ino);
>  	if (net == NULL)
>  		return -ENXIO;
> @@ -83,6 +96,11 @@ EXPORT_SYMBOL_GPL(single_open_net);
>  int seq_release_net(struct inode *ino, struct file *f)
>  {
>  	struct seq_file *seq;
> +	char *n;
> +
> +	n = d_path(&f->f_path, last_procfs_close, sizeof(last_procfs_close));
> +	if (!IS_ERR(n))
> +		memmove(last_procfs_close, n, strlen(n) + 1);
>  
>  	seq = f->private_data;
>  
> --


---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: patchwork.kernel.org down

[Roland Dreier]

 > Let's not mislead the reader of a crash dump, please.  How about:

 > +	printk(KERN_EMERG "last /proc..net open:  %s\n", last_procfs_open);
 > +	printk(KERN_EMERG "last /proc..net close: %s\n", last_procfs_close);

Fair enough... to be clear I wasn't proposing this patch be merged
anywhere -- it's just for debugging this kernel.org crash.'

Re: patchwork.kernel.org down

[Roland Dreier]

 >  > More excitement from https://bugzilla.kernel.org/show_bug.cgi?id=20702
 >  > 
 >  > Anyone want to take a stab at it?  I know I'd be appreciative.  I am
 >  > running a debug kernel with everything I could find and enable I even
 >  > remotely thought might prove helpful.  Nothing has jumped out yet though.
 > 
 > A little late, but perhaps running with the patch below might help us
 > make a bit of progress.  The idea is to dump the last /proc/net file
 > opened and closed, so we can at least have a clue as to where the crash
 > is coming from.  This should add lines like 
 > 
 >     last procfs open:  /proc/2443/net/arp
 >     last procfs close: /proc/2443/net/arp
 > 
 > to the oops output, so maybe we can zero in on things after you get a
 > crash with this applied.

Just curious -- did you get a chance to run with this patch applied?
Did you collect any oopses yet?

 - R.

Re: patchwork.kernel.org down

[J.H.]

On 01/10/2011 10:52 AM, Roland Dreier wrote:
>  >  > More excitement from https://bugzilla.kernel.org/show_bug.cgi?id=20702
>  >  > 
>  >  > Anyone want to take a stab at it?  I know I'd be appreciative.  I am
>  >  > running a debug kernel with everything I could find and enable I even
>  >  > remotely thought might prove helpful.  Nothing has jumped out yet though.
>  > 
>  > A little late, but perhaps running with the patch below might help us
>  > make a bit of progress.  The idea is to dump the last /proc/net file
>  > opened and closed, so we can at least have a clue as to where the crash
>  > is coming from.  This should add lines like 
>  > 
>  >     last procfs open:  /proc/2443/net/arp
>  >     last procfs close: /proc/2443/net/arp
>  > 
>  > to the oops output, so maybe we can zero in on things after you get a
>  > crash with this applied.
> 
> Just curious -- did you get a chance to run with this patch applied?
> Did you collect any oopses yet?

Not yet, I'm adding it in this afternoon.  I did the upgrade to Fedora
14 (and thus 2.6.35.10) and seem to be getting, if not the same bug, a
darned similar one:

http://pastebin.osuosl.org/36644

I haven't added it to 20702 yet as I haven't had time to confirm if it's
the same issue, but at first glance it looks it.  My plan is to compile
up a mainline kernel, with the above patch, and with debugging turned on
and to capture as much as I can from that.

- John 'Warthog9' Hawley

Re: patchwork.kernel.org down

[Roland Dreier]

 > Not yet, I'm adding it in this afternoon.  I did the upgrade to Fedora
 > 14 (and thus 2.6.35.10) and seem to be getting, if not the same bug, a
 > darned similar one:
 > 
 > http://pastebin.osuosl.org/36644

Actually that looks pretty different; the trace is

 put_ldisc+0x8d/0xb4
 tty_ldisc_reinit+0x43/0x5c
 tty_ldisc_hangup+0x10d/0x19d
 do_tty_hangup+0x108/0x343
 ? __raw_local_irq_save+0x1d/0x23
 tty_vhangup_self+0x27/0x34
 sys_vhangup+0x22/0x29
 system_call_fastpath+0x16/0x1b

which doesn't involve any /proc/net file being closed at all.

Hmm...

Re: patchwork.kernel.org down

[J.H.]

On 01/10/2011 12:33 PM, Roland Dreier wrote:
>  > Not yet, I'm adding it in this afternoon.  I did the upgrade to Fedora
>  > 14 (and thus 2.6.35.10) and seem to be getting, if not the same bug, a
>  > darned similar one:
>  > 
>  > http://pastebin.osuosl.org/36644
> 
> Actually that looks pretty different; the trace is
> 
>  put_ldisc+0x8d/0xb4
>  tty_ldisc_reinit+0x43/0x5c
>  tty_ldisc_hangup+0x10d/0x19d
>  do_tty_hangup+0x108/0x343
>  ? __raw_local_irq_save+0x1d/0x23
>  tty_vhangup_self+0x27/0x34
>  sys_vhangup+0x22/0x29
>  system_call_fastpath+0x16/0x1b
> 
> which doesn't involve any /proc/net file being closed at all.
> 
> Hmm...

My thoughts on why it was possibly similar is that it happens

[  793.932076] kernel BUG at mm/slub.c:2834!

where as 20702 is

kernel BUG at mm/slub.c:2835!

the rest didn't line up, and like I said I haven't specifically dug into it.

As a note I did end up with a full panic in the last hour, but nothing
got recorded (sadly).  I'm working on getting the kernel compiled up
with everything now so hopefully I'll start getting some additional data
to share.

- John 'Warthog9' Hawley

Re: patchwork.kernel.org down

[Roland Dreier]

 > I haven't added it to 20702 yet as I haven't had time to confirm if it's
 > the same issue, but at first glance it looks it.  My plan is to compile
 > up a mainline kernel, with the above patch, and with debugging turned on
 > and to capture as much as I can from that.

By the way, it is probably a good idea to boot with "slub_debug=FZP" to
make sure that slub debugging is turned on at runtime (unless you're
building at kernel with CONFIG_SLUB_DEBUG_ON=y, which I'm sure Fedora
doesn't do).  The crashes you're seeing definitely look like corruption
in the allocator.

 - R.