cleanerd segfault

cleanerd segfault

[David Arendt]

Hi,

I am using latest nilfs2 module and nilfs2 utils from today and I am
getting a segfault when trying to run cleanerd.

nilfs_cleanerd[5784]: segfault at b7f5b000 ip 0804a0db sp bff79720 error
4 in nilfs_cleanerd[8048000+5000]

Could you please look into this ?

Bye,
David Arendt

Re: cleanerd segfault

[Ryusuke Konishi]

Hi David,
On Thu, 05 Feb 2009 07:01:25 +0100, David Arendt wrote:
> Hi,
> 
> I am using latest nilfs2 module and nilfs2 utils from today and I am
> getting a segfault when trying to run cleanerd.
>
> nilfs_cleanerd[5784]: segfault at b7f5b000 ip 0804a0db sp bff79720 error
> 4 in nilfs_cleanerd[8048000+5000]
> 
> Could you please look into this ?

Thanks for reports!

You mean git tree versions?
I'm considering to make a minor release in a few days.
If the latest version on git trees seems unstable, I'd like to fix it
before that.

Regards,
Ryusuke

Re: cleanerd segfault

[admin-/LHdS3kC8BfYtjvyW6yDsg]

Yes the git versions from today.I think it must have been some problem introduced the during 2 months as 2 month ago cleanerd run perfectly stable. I have this problem on different nilfs2 filesystems. I am now creating a fresh nilfs2 filesystem to check if I will have the same segfault there and report back. The kernel version is 2.6.28.3 32bit.

Thanks in advance
Bye,
David Arendt

- original message -
Subject:	Re: [NILFS users] cleanerd segfault
From:	Ryusuke Konishi <ryusuke-sG5X7nlA6pw@public.gmane.org>
Date:		05/02/2009 06:54

Hi David,
On Thu, 05 Feb 2009 07:01:25 +0100, David Arendt wrote:
> Hi,
> 
> I am using latest nilfs2 module and nilfs2 utils from today and I am
> getting a segfault when trying to run cleanerd.
>
> nilfs_cleanerd[5784]: segfault at b7f5b000 ip 0804a0db sp bff79720 error
> 4 in nilfs_cleanerd[8048000+5000]
> 
> Could you please look into this ?

Thanks for reports!

You mean git tree versions?
I'm considering to make a minor release in a few days.
If the latest version on git trees seems unstable, I'd like to fix it
before that.

Regards,
Ryusuke

Re: cleanerd segfault

[admin-/LHdS3kC8BfYtjvyW6yDsg]

I can confirm that it is also reproducable on a newly created nilfs2 populated with some files.

- original message -
Subject:	Re: [NILFS users] cleanerd segfault
From:	Ryusuke Konishi <ryusuke-sG5X7nlA6pw@public.gmane.org>
Date:		05/02/2009 06:54

Hi David,
On Thu, 05 Feb 2009 07:01:25 +0100, David Arendt wrote:
> Hi,
> 
> I am using latest nilfs2 module and nilfs2 utils from today and I am
> getting a segfault when trying to run cleanerd.
>
> nilfs_cleanerd[5784]: segfault at b7f5b000 ip 0804a0db sp bff79720 error
> 4 in nilfs_cleanerd[8048000+5000]
> 
> Could you please look into this ?

Thanks for reports!

You mean git tree versions?
I'm considering to make a minor release in a few days.
If the latest version on git trees seems unstable, I'd like to fix it
before that.

Regards,
Ryusuke

Re: cleanerd segfault

[Ryusuke Konishi]

On Thu,  5 Feb 2009 08:52:32 +0100, admin-/LHdS3kC8BfYtjvyW6yDsg@public.gmane.org wrote:
> I can confirm that it is also reproducable on a newly created nilfs2
> populated with some files.

All right.
Could you try the following patch against nilfs2-utils ?

This makes cleanerd which can dump backtrace to /var/log/segv_xxx.

To enable the feature, you need gdb and a 'backtrace' script
available on http://samba.org/ftp/unpacked/junkcode/segv_handler/


Regards,
Ryusuke

--
diff --git a/sbin/cleanerd/Makefile.am b/sbin/cleanerd/Makefile.am
index e6b2553..488518e 100644
--- a/sbin/cleanerd/Makefile.am
+++ b/sbin/cleanerd/Makefile.am
@@ -8,7 +8,7 @@ sbin_PROGRAMS = nilfs_cleanerd
 
 nilfs_cleanerd_SOURCES = cleanerd.c cldconfig.c vector.c \
 	cleanerd.h cldconfig.h vector.h
-nilfs_cleanerd_CFLAGS = -Wall
+nilfs_cleanerd_CFLAGS = -Wall -g
 nilfs_cleanerd_CPPFLAGS = -I$(top_srcdir)/include \
 	-DSYSCONFDIR=\"$(sysconfdir)\" -D_GNU_SOURCE
 nilfs_cleanerd_LDADD = $(top_builddir)/lib/libnilfs.la
diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index cae22be..5f67e51 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -829,6 +829,46 @@ static int set_sighup_handler(void)
 	return sigaction(SIGHUP, &act, NULL);
 }
 
+/*
+ * The following part is based on segv_handler by Andrew Tridgell
+ * found at http://samba.org/ftp/unpacked/junkcode/segv_handler/
+ *
+ * To enable this feature, install gdb and 'backtrace' script available
+ * on the above site.
+ */
+static RETSIGTYPE handle_segv(int signum)
+{
+	char cmd[100];
+	char progname[100];
+	char *p;
+	int n;
+
+	n = readlink("/proc/self/exe",progname,sizeof(progname));
+	progname[n] = 0;
+
+	p = strrchr(progname, '/');
+	*p = 0;
+	
+	snprintf(cmd, sizeof(cmd),
+		 "backtrace %d > /var/log/segv_%s.%d.out 2>&1", 
+		 (int)getpid(), p+1, (int)getpid());
+	system(cmd);
+	signal(signum, SIG_DFL);
+}
+
+static int set_segv_handler(void)
+{
+	struct sigaction act;
+
+	act.sa_handler = handle_segv;
+	sigfillset(&act.sa_mask);
+	act.sa_flags = 0;
+	return sigaction(SIGSEGV, &act, NULL) || sigaction(SIGBUS, &act, NULL);
+}
+/*
+ * End of segv_handler
+ */
+
 #define timeval_to_timespec(tv, ts)		\
 do {						\
 	(ts)->tv_sec = (tv)->tv_sec;		\
@@ -859,6 +899,10 @@ static int nilfs_cleanerd_clean_loop(struct nilfs_cleanerd *cleanerd)
 		syslog(LOG_ERR, "cannot set SIGHUP signal handler: %m");
 		return -1;
 	}
+	if (set_segv_handler() < 0) {
+		syslog(LOG_ERR, "cannot set SIGSEGV signal handler: %m");
+		return -1;
+	}
 
 	nilfs_cleanerd_reload_config = 0;
 
-- 
1.5.6.5

Re: cleanerd segfault

[Ryusuke Konishi]

Hi David,
On Thu, 05 Feb 2009 18:19:04 +0900 (JST), Ryusuke Konishi wrote:
> On Thu,  5 Feb 2009 08:52:32 +0100, admin-/LHdS3kC8BfYtjvyW6yDsg@public.gmane.org wrote:
> > I can confirm that it is also reproducable on a newly created nilfs2
> > populated with some files.
> 
> All right.
> Could you try the following patch against nilfs2-utils ?
> 
> This makes cleanerd which can dump backtrace to /var/log/segv_xxx.
> 
> To enable the feature, you need gdb and a 'backtrace' script
> available on http://samba.org/ftp/unpacked/junkcode/segv_handler/
> 
> 
> Regards,
> Ryusuke

I've pushed out updates that would fix this problem.
(Both nilfs-utils and nilfs2-modules were updated).

It turned out to be a regression brought by one of the recent bug
fixes.  It was lucky that this was found before the next release.

David, thank you for your many feedbacks.

Regards,
Ryusuke Konishi

Re: cleanerd segfault

[Ryusuke Konishi]

On Sat, 07 Feb 2009 20:38:08 +0900 (JST), Ryusuke Konishi wrote:
> I've pushed out updates that would fix this problem.
> (Both nilfs-utils and nilfs2-modules were updated).
> 
> It turned out to be a regression brought by one of the recent bug
> fixes.  It was lucky that this was found before the next release.

Grr, I found another critical bug on a checkpoint ioctl.
It may be the cause of the GC related troubles.

Please wait trying this until the next patch will be pushed out.

Regards,
Ryusuke