Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[Jeff Layton]

On Mon, 02 Aug 2010 16:47:52 +0900
Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> wrote:

> Hello.
> 
> I got below failure on Debian Sarge when starting /usr/sbin/rpc.nfsd .
> 2.6.35 works fine.
> Kernel config is at http://I-love.SAKURA.ne.jp/tmp/config-2.6.35-next-20100802
> Regards.
> 
> 
> [   26.081814] pcnet32 0000:02:00.0: eth0: link up
> [   36.349815] BUG: unable to handle kernel NULL pointer dereference at 0000002c
> [   36.351254] IP: [<c11455a6>] nfsd_svc+0x56/0x110
> [   36.351398] *pde = 00000000 
> [   36.351398] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
> [   36.351398] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/host0/target0:0:1/0:0:1:0/type
> [   36.351398] Modules linked in: pcnet32
> [   36.351398] 
> [   36.351398] Pid: 2615, comm: rpc.nfsd Tainted: G        W   2.6.35-next-20100802 #2 440BX Desktop Reference Platform/VMware Virtual Platform
> [   36.351398] EIP: 0060:[<c11455a6>] EFLAGS: 00010202 CPU: 0
> [   36.351398] EIP is at nfsd_svc+0x56/0x110
> [   36.351398] EAX: 00000000 EBX: 00000008 ECX: 00000000 EDX: c154c728
> [   36.351398] ESI: 00000000 EDI: 00000801 EBP: dcf3bf68 ESP: dcf3bf54
> [   36.351398]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
> [   36.351398] Process rpc.nfsd (pid: 2615, ti=dcf3b000 task=de6a8130 task.ti=dcf3b000)
> [   36.351398] Stack:
> [   36.351398]  dec3cf28 00f3bf70 00000002 dec3cf28 00000008 dcf3bf70 c1145bba dcf3bf84
> [   36.351398] <0> c1145abf c1393f40 dec3cf28 00000000 dcf3bfac c10f633b dec3cf6c dec3cf6c
> [   36.351398] <0> 00000000 bfb34204 00000201 00000000 b7740b90 bfb3420c dcf3b000 c137fba1
> [   36.351398] Call Trace:
> [   36.351398]  [<c1145bba>] ? write_svc+0x1a/0x30
> [   36.351398]  [<c1145abf>] ? nfsctl_transaction_write+0x5f/0x80
> [   36.351398]  [<c10f633b>] ? sys_nfsservctl+0xab/0xf0
> [   36.351398]  [<c137fba1>] ? syscall_call+0x7/0xb
> [   36.351398] Code: 00 00 00 0f 4e d8 81 fb 01 20 00 00 b8 00 20 00 00 0f 4d d8 31 f6 85 db 0f 85 97 00 00 00 a1 84 95 c9 c1 85 c0 74 69 c6 45 f3 00 <8b> 48 2c 85 c9 75 13 85 db 74 0f c6 45 f3 01 8d 74 26 00 8d bc 
> [   36.351398] EIP: [<c11455a6>] nfsd_svc+0x56/0x110 SS:ESP 0068:dcf3bf54
> [   36.351398] CR2: 000000000000002c
> [   36.397072] ---[ end trace 3ca898c1e9981f94 ]---
> [   37.597439] NET: Registered protocol family 10
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

(cc'ing linux-nfs list...)

Ahh I think I see the bug, nfsd_svc does this:

        first_thread = (nfsd_serv->sv_nrthreads == 0) && (nrservs != 0);

...and only later does this:

        error = nfsd_create_serv();
        if (error)
                goto out_shutdown;

Because you're using the older nfsctl interface rather
than /proc/fs/nfsd, nfsd_svc is called before write_versions and
nfsd_serv is NULL.

Does the following patch fix it?

diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 92173bd..79cfd7a 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -432,7 +432,9 @@ nfsd_svc(unsigned short port, int nrservs)
 	if (nrservs == 0 && nfsd_serv == NULL)
 		goto out;
 
-	first_thread = (nfsd_serv->sv_nrthreads == 0) && (nrservs != 0);
+	first_thread = ((nfsd_serv == NULL) ||
+			(nfsd_serv->sv_nrthreads == 0)) &&
+		       (nrservs != 0);
 
 	if (first_thread) {
 		error = nfsd_startup(port, nrservs);


-- 
Jeff Layton <jlayton@redhat.com>

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[Jeff Layton]

On Mon, 2 Aug 2010 10:32:14 -0400
Jeff Layton <jlayton@redhat.com> wrote:

> On Mon, 02 Aug 2010 16:47:52 +0900
> Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> wrote:
> 
> > Hello.
> > 
> > I got below failure on Debian Sarge when starting /usr/sbin/rpc.nfsd .
> > 2.6.35 works fine.
> > Kernel config is at http://I-love.SAKURA.ne.jp/tmp/config-2.6.35-next-20100802
> > Regards.
> > 
> > 
> > [   26.081814] pcnet32 0000:02:00.0: eth0: link up
> > [   36.349815] BUG: unable to handle kernel NULL pointer dereference at 0000002c
> > [   36.351254] IP: [<c11455a6>] nfsd_svc+0x56/0x110
> > [   36.351398] *pde = 00000000 
> > [   36.351398] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
> > [   36.351398] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/host0/target0:0:1/0:0:1:0/type
> > [   36.351398] Modules linked in: pcnet32
> > [   36.351398] 
> > [   36.351398] Pid: 2615, comm: rpc.nfsd Tainted: G        W   2.6.35-next-20100802 #2 440BX Desktop Reference Platform/VMware Virtual Platform
> > [   36.351398] EIP: 0060:[<c11455a6>] EFLAGS: 00010202 CPU: 0
> > [   36.351398] EIP is at nfsd_svc+0x56/0x110
> > [   36.351398] EAX: 00000000 EBX: 00000008 ECX: 00000000 EDX: c154c728
> > [   36.351398] ESI: 00000000 EDI: 00000801 EBP: dcf3bf68 ESP: dcf3bf54
> > [   36.351398]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
> > [   36.351398] Process rpc.nfsd (pid: 2615, ti=dcf3b000 task=de6a8130 task.ti=dcf3b000)
> > [   36.351398] Stack:
> > [   36.351398]  dec3cf28 00f3bf70 00000002 dec3cf28 00000008 dcf3bf70 c1145bba dcf3bf84
> > [   36.351398] <0> c1145abf c1393f40 dec3cf28 00000000 dcf3bfac c10f633b dec3cf6c dec3cf6c
> > [   36.351398] <0> 00000000 bfb34204 00000201 00000000 b7740b90 bfb3420c dcf3b000 c137fba1
> > [   36.351398] Call Trace:
> > [   36.351398]  [<c1145bba>] ? write_svc+0x1a/0x30
> > [   36.351398]  [<c1145abf>] ? nfsctl_transaction_write+0x5f/0x80
> > [   36.351398]  [<c10f633b>] ? sys_nfsservctl+0xab/0xf0
> > [   36.351398]  [<c137fba1>] ? syscall_call+0x7/0xb
> > [   36.351398] Code: 00 00 00 0f 4e d8 81 fb 01 20 00 00 b8 00 20 00 00 0f 4d d8 31 f6 85 db 0f 85 97 00 00 00 a1 84 95 c9 c1 85 c0 74 69 c6 45 f3 00 <8b> 48 2c 85 c9 75 13 85 db 74 0f c6 45 f3 01 8d 74 26 00 8d bc 
> > [   36.351398] EIP: [<c11455a6>] nfsd_svc+0x56/0x110 SS:ESP 0068:dcf3bf54
> > [   36.351398] CR2: 000000000000002c
> > [   36.397072] ---[ end trace 3ca898c1e9981f94 ]---
> > [   37.597439] NET: Registered protocol family 10
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> (cc'ing linux-nfs list...)
> 
> Ahh I think I see the bug, nfsd_svc does this:
> 
>         first_thread = (nfsd_serv->sv_nrthreads == 0) && (nrservs != 0);
> 
> ...and only later does this:
> 
>         error = nfsd_create_serv();
>         if (error)
>                 goto out_shutdown;
> 
> Because you're using the older nfsctl interface rather
> than /proc/fs/nfsd, nfsd_svc is called before write_versions and
> nfsd_serv is NULL.
> 
> Does the following patch fix it?
> 
> diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
> index 92173bd..79cfd7a 100644
> --- a/fs/nfsd/nfssvc.c
> +++ b/fs/nfsd/nfssvc.c
> @@ -432,7 +432,9 @@ nfsd_svc(unsigned short port, int nrservs)
>  	if (nrservs == 0 && nfsd_serv == NULL)
>  		goto out;
>  
> -	first_thread = (nfsd_serv->sv_nrthreads == 0) && (nrservs != 0);
> +	first_thread = ((nfsd_serv == NULL) ||
> +			(nfsd_serv->sv_nrthreads == 0)) &&
> +		       (nrservs != 0);
>  
>  	if (first_thread) {
>  		error = nfsd_startup(port, nrservs);
> 
> 

nevermind...that patch will probably fix this panic, but there's another
possible one in nfsd_init_socks. We'll have to fix that one too.

-- 
Jeff Layton <jlayton@redhat.com>

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Mon, Aug 02, 2010 at 10:36:20AM -0400, Jeff Layton wrote:
> nevermind...that patch will probably fix this panic, but there's another
> possible one in nfsd_init_socks. We'll have to fix that one too.

(After private conversation with Jeff): something like this?
Compile-tested only.

--b.

commit 86d0cc3b91315c475c1c38ee7a06b5ebe5c01755
Author: J. Bruce Fields <bfields@redhat.com>
Date:   Mon Aug 2 14:12:44 2010 -0400

    nfsd: fix startup/shutdown order bug
    
    We must create the server before we can call init_socks or check the
    number of threads.
    
    Symptoms were a NULL pointer dereference in nfsd_svc().  Problem
    identified by Jeff Layton.
    
    Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
    Signed-off-by: J. Bruce Fields <bfields@redhat.com>

diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 92173bd..1de1cb3 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -432,29 +432,30 @@ nfsd_svc(unsigned short port, int nrservs)
 	if (nrservs == 0 && nfsd_serv == NULL)
 		goto out;
 
+	error = nfsd_create_serv();
+	if (error)
+		goto out;
+
 	first_thread = (nfsd_serv->sv_nrthreads == 0) && (nrservs != 0);
 
 	if (first_thread) {
 		error = nfsd_startup(port, nrservs);
 		if (error)
-			goto out;
+			goto out_destroy;
 	}
-	error = nfsd_create_serv();
-	if (error)
-		goto out_shutdown;
 	error = svc_set_num_threads(nfsd_serv, NULL, nrservs);
 	if (error)
-		goto out_destroy;
+		goto out_shutdown;
 	/* We are holding a reference to nfsd_serv which
 	 * we don't want to count in the return value,
 	 * so subtract 1
 	 */
 	error = nfsd_serv->sv_nrthreads - 1;
-out_destroy:
-	svc_destroy(nfsd_serv);		/* Release server */
 out_shutdown:
 	if (error < 0 && first_thread)
 		nfsd_shutdown();
+out_destroy:
+	svc_destroy(nfsd_serv);		/* Release server */
 out:
 	mutex_unlock(&nfsd_mutex);
 	return error;

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[Jeff Layton]

On Mon, 2 Aug 2010 14:16:34 -0400
"J. Bruce Fields" <bfields@fieldses.org> wrote:

> On Mon, Aug 02, 2010 at 10:36:20AM -0400, Jeff Layton wrote:
> > nevermind...that patch will probably fix this panic, but there's another
> > possible one in nfsd_init_socks. We'll have to fix that one too.
> 
> (After private conversation with Jeff): something like this?
> Compile-tested only.
> 
> --b.
> 
> commit 86d0cc3b91315c475c1c38ee7a06b5ebe5c01755
> Author: J. Bruce Fields <bfields@redhat.com>
> Date:   Mon Aug 2 14:12:44 2010 -0400
> 
>     nfsd: fix startup/shutdown order bug
>     
>     We must create the server before we can call init_socks or check the
>     number of threads.
>     
>     Symptoms were a NULL pointer dereference in nfsd_svc().  Problem
>     identified by Jeff Layton.
>     
>     Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
>     Signed-off-by: J. Bruce Fields <bfields@redhat.com>
> 
> diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
> index 92173bd..1de1cb3 100644
> --- a/fs/nfsd/nfssvc.c
> +++ b/fs/nfsd/nfssvc.c
> @@ -432,29 +432,30 @@ nfsd_svc(unsigned short port, int nrservs)
>  	if (nrservs == 0 && nfsd_serv == NULL)
>  		goto out;
>  
> +	error = nfsd_create_serv();
> +	if (error)
> +		goto out;
> +
>  	first_thread = (nfsd_serv->sv_nrthreads == 0) && (nrservs != 0);
>  
>  	if (first_thread) {
>  		error = nfsd_startup(port, nrservs);
>  		if (error)
> -			goto out;
> +			goto out_destroy;
>  	}
> -	error = nfsd_create_serv();
> -	if (error)
> -		goto out_shutdown;
>  	error = svc_set_num_threads(nfsd_serv, NULL, nrservs);
>  	if (error)
> -		goto out_destroy;
> +		goto out_shutdown;
>  	/* We are holding a reference to nfsd_serv which
>  	 * we don't want to count in the return value,
>  	 * so subtract 1
>  	 */
>  	error = nfsd_serv->sv_nrthreads - 1;
> -out_destroy:
> -	svc_destroy(nfsd_serv);		/* Release server */
>  out_shutdown:
>  	if (error < 0 && first_thread)
>  		nfsd_shutdown();
> +out_destroy:
> +	svc_destroy(nfsd_serv);		/* Release server */
>  out:
>  	mutex_unlock(&nfsd_mutex);
>  	return error;

I was able to reproduce the problem and the patch fixes it. Assuming
that Tetsuo's testing goes well:

Reviewed-and-Tested-by: Jeff Layton <jlayton@redhat.com>

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[Tetsuo Handa]

Hello.

That patch solved the NULL pointer dereference problem. Thank you.

But I got another problem. After applying that patch on 2.6.35-next-20100802 ,
mount operation fails with timeout error.

# cat /etc/exports
/usr/src/ *(rw,no_root_squash,async)
# time mount 127.0.0.1:/usr/src/ /mnt/
mount: Connection timed out

real    1m21.099s
user    0m0.000s
sys     0m0.028s


2.6.35 works fine. (shown below)

# time mount 127.0.0.1:/usr/src/ /mnt/

real    0m0.105s
user    0m0.000s
sys     0m0.020s

Regards.

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Tue, Aug 03, 2010 at 10:09:03AM +0900, Tetsuo Handa wrote:
> Hello.
> 
> That patch solved the NULL pointer dereference problem. Thank you.
> 
> But I got another problem. After applying that patch on 2.6.35-next-20100802 ,
> mount operation fails with timeout error.

Argh, yes, problem found, I think--I'll do some more testing and send
you another attempt....

--b.

> 
> # cat /etc/exports
> /usr/src/ *(rw,no_root_squash,async)
> # time mount 127.0.0.1:/usr/src/ /mnt/
> mount: Connection timed out
> 
> real    1m21.099s
> user    0m0.000s
> sys     0m0.028s
> 
> 
> 2.6.35 works fine. (shown below)
> 
> # time mount 127.0.0.1:/usr/src/ /mnt/
> 
> real    0m0.105s
> user    0m0.000s
> sys     0m0.020s
> 
> Regards.

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Tue, Aug 03, 2010 at 11:48:51AM -0400, J. Bruce Fields wrote:
> On Tue, Aug 03, 2010 at 10:09:03AM +0900, Tetsuo Handa wrote:
> > Hello.
> > 
> > That patch solved the NULL pointer dereference problem. Thank you.
> > 
> > But I got another problem. After applying that patch on 2.6.35-next-20100802 ,
> > mount operation fails with timeout error.
> 
> Argh, yes, problem found, I think--I'll do some more testing and send
> you another attempt....

How about this?

(By the way, are you using something other than the standard
/etc/init.d/nfs-kernel-server to start/stop the server?  Or have you
customized your installation in any way?  Just curious, as the bugs
you're finding are good, but I'd expect different symptoms from the
default setup.)

--b.

commit 3deb279d6e5625407919a875db3a2461199566b3
Author: J. Bruce Fields <bfields@redhat.com>
Date:   Mon Aug 2 14:12:44 2010 -0400

    nfsd: fix startup/shutdown order bug
    
    We must create the server before we can call init_socks or check the
    number of threads.
    
    Symptoms were a NULL pointer dereference in nfsd_svc().  Problem
    identified by Jeff Layton.
    
    Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
    Signed-off-by: J. Bruce Fields <bfields@redhat.com>

diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 92173bd..58e3d4c 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -420,7 +420,7 @@ int
 nfsd_svc(unsigned short port, int nrservs)
 {
 	int	error;
-	bool	first_thread;
+	bool	unstarted, first_thread;
 
 	mutex_lock(&nfsd_mutex);
 	dprintk("nfsd: creating service\n");
@@ -432,29 +432,31 @@ nfsd_svc(unsigned short port, int nrservs)
 	if (nrservs == 0 && nfsd_serv == NULL)
 		goto out;
 
-	first_thread = (nfsd_serv->sv_nrthreads == 0) && (nrservs != 0);
+	unstarted = nfsd_serv == NULL || nfsd_serv->sv_nrthreads == 0;
+	first_thread = unstarted && (nrservs != 0);
+
+	error = nfsd_create_serv();
+	if (error)
+		goto out;
 
 	if (first_thread) {
 		error = nfsd_startup(port, nrservs);
 		if (error)
-			goto out;
+			goto out_destroy;
 	}
-	error = nfsd_create_serv();
-	if (error)
-		goto out_shutdown;
 	error = svc_set_num_threads(nfsd_serv, NULL, nrservs);
 	if (error)
-		goto out_destroy;
+		goto out_shutdown;
 	/* We are holding a reference to nfsd_serv which
 	 * we don't want to count in the return value,
 	 * so subtract 1
 	 */
 	error = nfsd_serv->sv_nrthreads - 1;
-out_destroy:
-	svc_destroy(nfsd_serv);		/* Release server */
 out_shutdown:
 	if (error < 0 && first_thread)
 		nfsd_shutdown();
+out_destroy:
+	svc_destroy(nfsd_serv);		/* Release server */
 out:
 	mutex_unlock(&nfsd_mutex);
 	return error;

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[Tetsuo Handa]

J. Bruce Fields wrote:
> How about this?

After reverting commit 86d0cc3b91315c475c1c38ee7a06b5ebe5c01755 and applying
commit 3deb279d6e5625407919a875db3a2461199566b3, I get below NULL pointer
dereference problem when doing "mount 127.0.0.1:/usr/src/ /mnt/".

[   96.398495] BUG: unable to handle kernel NULL pointer dereference at 00000010
[   96.400348] IP: [<c1356dd4>] svc_process_common+0x2c4/0x5c0
[   96.401606] *pde = 00000000 
[   96.401606] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
[   96.401606] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/host0/target0:0:1/0:0:1:0/type
[   96.401606] Modules linked in: nfs ipv6 pcnet32
[   96.401606] 
[   96.401606] Pid: 2623, comm: nfsd Tainted: G        W   2.6.35-next-20100802 #4 440BX Desktop Reference Platform/VMware Virtual Platform
[   96.401606] EIP: 0060:[<c1356dd4>] EFLAGS: 00010246 CPU: 1
[   96.401606] EIP is at svc_process_common+0x2c4/0x5c0
[   96.401606] EAX: 00000000 EBX: dfb8b0c8 ECX: 00000001 EDX: 00000004
[   96.401606] ESI: dfb8b0f0 EDI: 00000010 EBP: dcac4f40 ESP: dcac4ef0
[   96.401606]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[   96.401606] Process nfsd (pid: 2623, ti=dcac4000 task=dc85a7b0 task.ti=dcac4000)
[   96.401606] Stack:
[   96.401606]  dc85a7b0 00000002 00000000 dc85ac5c dc85a7b0 dcac4f28 00000004 00000000
[   96.401606] <0> 00000002 000186a3 dcb6c008 dcb6c014 dc805f30 c154ca20 c154cca8 dfb8b000
[   96.401606] <0> 01000000 8c4194fd dfb8b000 dfb8b0c8 dcac4f68 c13571cf dcac4f68 c105d087
[   96.401606] Call Trace:
[   96.401606]  [<c13571cf>] ? svc_process+0xff/0x110
[   96.401606]  [<c105d087>] ? __validate_process_creds+0x47/0xd0
[   96.401606]  [<c1145739>] ? nfsd+0xc9/0x160
[   96.401606]  [<c1035f86>] ? complete+0x46/0x60
[   96.401606]  [<c1055d05>] ? kthread+0x75/0x80
[   96.401606]  [<c1145670>] ? nfsd+0x0/0x160
[   96.401606]  [<c1055c90>] ? kthread+0x0/0x80
[   96.401606]  [<c100317a>] ? kernel_thread_helper+0x6/0x1c
[   96.401606] Code: 4d dc c7 01 00 00 00 00 8b 55 e4 83 46 04 04 8b 42 10 ff 42 18 8b 4d ec 8b b9 6c 0d 00 00 89 45 c8 89 c1 c1 e9 02 31 c0 8b 55 c8 <f3> ab f6 c2 02 74 02 66 ab f6 c2 01 74 01 aa 8b 45 e4 8b 4d ec 
[   96.401606] EIP: [<c1356dd4>] svc_process_common+0x2c4/0x5c0 SS:ESP 0068:dcac4ef0
[   96.401606] CR2: 0000000000000010
[   96.527321] ---[ end trace 0de1e1ad73b15980 ]---

> (By the way, are you using something other than the standard
> /etc/init.d/nfs-kernel-server to start/stop the server?  Or have you
> customized your installation in any way?  Just curious, as the bugs
> you're finding are good, but I'd expect different symptoms from the
> default setup.)
I'm using standard /etc/init.d/nfs-kernel-server script installed by Debian
Sarge and using it without modification.

# ls -l /etc/init.d/nfs-*
-rwxr-xr-x  1 root root 1984 Jan  5  2005 /etc/init.d/nfs-common
-rwxr-xr-x  1 root root 2356 Aug  4  2003 /etc/init.d/nfs-kernel-server
-rwxr-xr-x  1 root root 1241 Jan 30  2006 /etc/init.d/nfs-user-server

Regards.

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Wed, Aug 04, 2010 at 09:13:48AM +0900, Tetsuo Handa wrote:
> J. Bruce Fields wrote:
> > How about this?
> 
> After reverting commit 86d0cc3b91315c475c1c38ee7a06b5ebe5c01755 and applying
> commit 3deb279d6e5625407919a875db3a2461199566b3, I get below NULL pointer
> dereference problem when doing "mount 127.0.0.1:/usr/src/ /mnt/".

OK, I'm not seeing the explanation yet.....

> 
> [   96.398495] BUG: unable to handle kernel NULL pointer dereference at 00000010
> [   96.400348] IP: [<c1356dd4>] svc_process_common+0x2c4/0x5c0

Maybe figuring out exactly hwere that is would help work out what's
going on.  Doing

	make net/sunrpc/svc.lst

then looking for c1356dd4 (or just mailing me svc.lst) could help.


> [   96.401606] *pde = 00000000 
> [   96.401606] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
> [   96.401606] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/host0/target0:0:1/0:0:1:0/type
> [   96.401606] Modules linked in: nfs ipv6 pcnet32
> [   96.401606] 
> [   96.401606] Pid: 2623, comm: nfsd Tainted: G        W   2.6.35-next-20100802 #4 440BX Desktop Reference Platform/VMware Virtual Platform
> [   96.401606] EIP: 0060:[<c1356dd4>] EFLAGS: 00010246 CPU: 1
> [   96.401606] EIP is at svc_process_common+0x2c4/0x5c0
> [   96.401606] EAX: 00000000 EBX: dfb8b0c8 ECX: 00000001 EDX: 00000004
> [   96.401606] ESI: dfb8b0f0 EDI: 00000010 EBP: dcac4f40 ESP: dcac4ef0
> [   96.401606]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> [   96.401606] Process nfsd (pid: 2623, ti=dcac4000 task=dc85a7b0 task.ti=dcac4000)
> [   96.401606] Stack:
> [   96.401606]  dc85a7b0 00000002 00000000 dc85ac5c dc85a7b0 dcac4f28 00000004 00000000
> [   96.401606] <0> 00000002 000186a3 dcb6c008 dcb6c014 dc805f30 c154ca20 c154cca8 dfb8b000
> [   96.401606] <0> 01000000 8c4194fd dfb8b000 dfb8b0c8 dcac4f68 c13571cf dcac4f68 c105d087
> [   96.401606] Call Trace:
> [   96.401606]  [<c13571cf>] ? svc_process+0xff/0x110
> [   96.401606]  [<c105d087>] ? __validate_process_creds+0x47/0xd0
> [   96.401606]  [<c1145739>] ? nfsd+0xc9/0x160
> [   96.401606]  [<c1035f86>] ? complete+0x46/0x60
> [   96.401606]  [<c1055d05>] ? kthread+0x75/0x80
> [   96.401606]  [<c1145670>] ? nfsd+0x0/0x160
> [   96.401606]  [<c1055c90>] ? kthread+0x0/0x80
> [   96.401606]  [<c100317a>] ? kernel_thread_helper+0x6/0x1c
> [   96.401606] Code: 4d dc c7 01 00 00 00 00 8b 55 e4 83 46 04 04 8b 42 10 ff 42 18 8b 4d ec 8b b9 6c 0d 00 00 89 45 c8 89 c1 c1 e9 02 31 c0 8b 55 c8 <f3> ab f6 c2 02 74 02 66 ab f6 c2 01 74 01 aa 8b 45 e4 8b 4d ec 
> [   96.401606] EIP: [<c1356dd4>] svc_process_common+0x2c4/0x5c0 SS:ESP 0068:dcac4ef0
> [   96.401606] CR2: 0000000000000010
> [   96.527321] ---[ end trace 0de1e1ad73b15980 ]---
> 
> > (By the way, are you using something other than the standard
> > /etc/init.d/nfs-kernel-server to start/stop the server?  Or have you
> > customized your installation in any way?  Just curious, as the bugs
> > you're finding are good, but I'd expect different symptoms from the
> > default setup.)
> I'm using standard /etc/init.d/nfs-kernel-server script installed by Debian
> Sarge and using it without modification.

OK, I wonder if Sarge didn't yet mount the nfsd filesystem on
/proc/fs/nfsd.

--b.

> 
> # ls -l /etc/init.d/nfs-*
> -rwxr-xr-x  1 root root 1984 Jan  5  2005 /etc/init.d/nfs-common
> -rwxr-xr-x  1 root root 2356 Aug  4  2003 /etc/init.d/nfs-kernel-server
> -rwxr-xr-x  1 root root 1241 Jan 30  2006 /etc/init.d/nfs-user-server
> 
> Regards.

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[Tetsuo Handa]

J. Bruce Fields wrote:
> Maybe figuring out exactly hwere that is would help work out what's
> going on.  Doing
> 
> 	make net/sunrpc/svc.lst
> 
> then looking for c1356dd4 (or just mailing me svc.lst) could help.

"make net/sunrpc/svc.lst" failed due to following error.

  BFD: Dwarf Error: Abbrev offset (3238007024) greater than or equal to .debug_abbrev size (1607).

Manual printk() debug reported that
rqstp->rq_argp == rqstp->rq_resp == ZERO_SIZE_PTR and
procp->pc_argsize == procp->pc_ressize == 4.

--- linux-2.6.35-next.orig/net/sunrpc/svc.c
+++ linux-2.6.35-next/net/sunrpc/svc.c
@@ -1084,6 +1084,11 @@ svc_process_common(struct svc_rqst *rqst
 	procp->pc_count++;

 	/* Initialize storage for argp and resp */
+	printk(KERN_INFO "rqstp=%p procp=%p\n", rqstp, procp);
+	printk(KERN_INFO "rqstp->rq_argp=%p procp->pc_argsize=%u\n",
+	       rqstp->rq_argp, procp->pc_argsize);
+	printk(KERN_INFO "rqstp->rq_resp=%p procp->pc_ressize=%u\n",
+	       rqstp->rq_resp, procp->pc_ressize);
 	memset(rqstp->rq_argp, 0, procp->pc_argsize);
 	memset(rqstp->rq_resp, 0, procp->pc_ressize);
 

[   37.669174] NET: Registered protocol family 10
[   38.080725] svc: failed to register lockdv1 RPC service (errno 97).
[  122.895707] rqstp=dcb91000 procp=c154ca20
[  122.896533] rqstp->rq_argp=00000010 procp->pc_argsize=4
[  122.897484] rqstp->rq_resp=00000010 procp->pc_ressize=4
[  122.898609] BUG: unable to handle kernel NULL pointer dereference at 00000010
[  122.899964] IP: [<c1356e80>] svc_process_common+0x370/0x640
[  122.900493] *pde = 00000000 
[  122.900493] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
[  122.923308] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/host0/target0:0:1/0:0:1:0/type
[  122.923308] Modules linked in: nfs ipv6 pcnet32

> OK, I wonder if Sarge didn't yet mount the nfsd filesystem on
> /proc/fs/nfsd.

According to /proc/mounts , the nfsd filesystem is not mounted on
/proc/fs/nfsd . But mounting it manually before starting nfsd did not help.

Regards.

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Thu, Aug 05, 2010 at 10:10:16AM +0900, Tetsuo Handa wrote:
> J. Bruce Fields wrote:
> > Maybe figuring out exactly hwere that is would help work out what's
> > going on.  Doing
> > 
> > 	make net/sunrpc/svc.lst
> > 
> > then looking for c1356dd4 (or just mailing me svc.lst) could help.
> 
> "make net/sunrpc/svc.lst" failed due to following error.
> 
>   BFD: Dwarf Error: Abbrev offset (3238007024) greater than or equal to .debug_abbrev size (1607).
> 
> Manual printk() debug reported that
> rqstp->rq_argp == rqstp->rq_resp == ZERO_SIZE_PTR and

Huh.  As far as I can tell that will only happen if you've not no nfsd
versions defined; how is that happening?

--b.

> procp->pc_argsize == procp->pc_ressize == 4.
> 
> --- linux-2.6.35-next.orig/net/sunrpc/svc.c
> +++ linux-2.6.35-next/net/sunrpc/svc.c
> @@ -1084,6 +1084,11 @@ svc_process_common(struct svc_rqst *rqst
>  	procp->pc_count++;
> 
>  	/* Initialize storage for argp and resp */
> +	printk(KERN_INFO "rqstp=%p procp=%p\n", rqstp, procp);
> +	printk(KERN_INFO "rqstp->rq_argp=%p procp->pc_argsize=%u\n",
> +	       rqstp->rq_argp, procp->pc_argsize);
> +	printk(KERN_INFO "rqstp->rq_resp=%p procp->pc_ressize=%u\n",
> +	       rqstp->rq_resp, procp->pc_ressize);
>  	memset(rqstp->rq_argp, 0, procp->pc_argsize);
>  	memset(rqstp->rq_resp, 0, procp->pc_ressize);
>  
> 
> [   37.669174] NET: Registered protocol family 10
> [   38.080725] svc: failed to register lockdv1 RPC service (errno 97).
> [  122.895707] rqstp=dcb91000 procp=c154ca20
> [  122.896533] rqstp->rq_argp=00000010 procp->pc_argsize=4
> [  122.897484] rqstp->rq_resp=00000010 procp->pc_ressize=4
> [  122.898609] BUG: unable to handle kernel NULL pointer dereference at 00000010
> [  122.899964] IP: [<c1356e80>] svc_process_common+0x370/0x640
> [  122.900493] *pde = 00000000 
> [  122.900493] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
> [  122.923308] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/host0/target0:0:1/0:0:1:0/type
> [  122.923308] Modules linked in: nfs ipv6 pcnet32
> 
> > OK, I wonder if Sarge didn't yet mount the nfsd filesystem on
> > /proc/fs/nfsd.
> 
> According to /proc/mounts , the nfsd filesystem is not mounted on
> /proc/fs/nfsd . But mounting it manually before starting nfsd did not help.
> 
> Regards.

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Thu, Aug 05, 2010 at 04:46:12PM -0400, J. Bruce Fields wrote:
> On Thu, Aug 05, 2010 at 10:10:16AM +0900, Tetsuo Handa wrote:
> > J. Bruce Fields wrote:
> > > Maybe figuring out exactly hwere that is would help work out what's
> > > going on.  Doing
> > > 
> > > 	make net/sunrpc/svc.lst
> > > 
> > > then looking for c1356dd4 (or just mailing me svc.lst) could help.
> > 
> > "make net/sunrpc/svc.lst" failed due to following error.
> > 
> >   BFD: Dwarf Error: Abbrev offset (3238007024) greater than or equal to .debug_abbrev size (1607).
> > 
> > Manual printk() debug reported that
> > rqstp->rq_argp == rqstp->rq_resp == ZERO_SIZE_PTR and
> 
> Huh.  As far as I can tell that will only happen if you've not no nfsd
> versions defined; how is that happening?

OK, I think it's another startup-order problem: depending on how things
are started up, sv_nrthreads may already be nonzero, causing us to skip
nfsd_reset_versions(), so that the loop in __svc_create() ends up
leaving xdrsize 0, and then the kmalloc's in svc_prepare_thread() assign
ZERO_SIZE_PTR.

I need to think a little more about what we should be doing here.

--b.

> 
> --b.
> 
> > procp->pc_argsize == procp->pc_ressize == 4.
> > 
> > --- linux-2.6.35-next.orig/net/sunrpc/svc.c
> > +++ linux-2.6.35-next/net/sunrpc/svc.c
> > @@ -1084,6 +1084,11 @@ svc_process_common(struct svc_rqst *rqst
> >  	procp->pc_count++;
> > 
> >  	/* Initialize storage for argp and resp */
> > +	printk(KERN_INFO "rqstp=%p procp=%p\n", rqstp, procp);
> > +	printk(KERN_INFO "rqstp->rq_argp=%p procp->pc_argsize=%u\n",
> > +	       rqstp->rq_argp, procp->pc_argsize);
> > +	printk(KERN_INFO "rqstp->rq_resp=%p procp->pc_ressize=%u\n",
> > +	       rqstp->rq_resp, procp->pc_ressize);
> >  	memset(rqstp->rq_argp, 0, procp->pc_argsize);
> >  	memset(rqstp->rq_resp, 0, procp->pc_ressize);
> >  
> > 
> > [   37.669174] NET: Registered protocol family 10
> > [   38.080725] svc: failed to register lockdv1 RPC service (errno 97).
> > [  122.895707] rqstp=dcb91000 procp=c154ca20
> > [  122.896533] rqstp->rq_argp=00000010 procp->pc_argsize=4
> > [  122.897484] rqstp->rq_resp=00000010 procp->pc_ressize=4
> > [  122.898609] BUG: unable to handle kernel NULL pointer dereference at 00000010
> > [  122.899964] IP: [<c1356e80>] svc_process_common+0x370/0x640
> > [  122.900493] *pde = 00000000 
> > [  122.900493] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
> > [  122.923308] last sysfs file: /sys/devices/pci0000:00/0000:00:10.0/host0/target0:0:1/0:0:1:0/type
> > [  122.923308] Modules linked in: nfs ipv6 pcnet32
> > 
> > > OK, I wonder if Sarge didn't yet mount the nfsd filesystem on
> > > /proc/fs/nfsd.
> > 
> > According to /proc/mounts , the nfsd filesystem is not mounted on
> > /proc/fs/nfsd . But mounting it manually before starting nfsd did not help.
> > 
> > Regards.

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[Tetsuo Handa]

I compared using below patch.

 fs/nfsd/nfssvc.c |   11 +++++++++++
 net/sunrpc/svc.c |   12 ++++++++++++
 2 files changed, 23 insertions(+)

--- linux-2.6.35-next.orig/fs/nfsd/nfssvc.c
+++ linux-2.6.35-next/fs/nfsd/nfssvc.c
@@ -263,15 +263,26 @@ void nfsd_reset_versions(void)
 	int found_one = 0;
 	int i;
 
+	printk(KERN_INFO "***** %s is called *****.\n", __func__);
 	for (i = NFSD_MINVERS; i < NFSD_NRVERS; i++) {
 		if (nfsd_program.pg_vers[i])
 			found_one = 1;
 	}
 
+	printk(KERN_INFO "***** found_one=%u *****.\n", found_one);
 	if (!found_one) {
+		printk(KERN_INFO
+		       "***** &nfsd_program=%p nfsd_version=%p *****.\n",
+		       &nfsd_program, nfsd_version);
+		printk(KERN_INFO
+		       "***** NFSD_MINVERS=%u NFSD_NRVERS=%u *****.\n",
+		       NFSD_MINVERS, NFSD_NRVERS);
 		for (i = NFSD_MINVERS; i < NFSD_NRVERS; i++)
 			nfsd_program.pg_vers[i] = nfsd_version[i];
 #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL)
+		printk(KERN_INFO
+		       "***** NFSD_ACL_MINVERS=%u NFSD_ACL_NRVERS=%u *****.\n",
+		       NFSD_ACL_MINVERS, NFSD_ACL_NRVERS);
 		for (i = NFSD_ACL_MINVERS; i < NFSD_ACL_NRVERS; i++)
 			nfsd_acl_program.pg_vers[i] =
 				nfsd_acl_version[i];
--- linux-2.6.35-next.orig/net/sunrpc/svc.c
+++ linux-2.6.35-next/net/sunrpc/svc.c
@@ -379,7 +379,9 @@ __svc_create(struct svc_program *prog, u
 	serv->sv_max_mesg  = roundup(serv->sv_max_payload + PAGE_SIZE, PAGE_SIZE);
 	serv->sv_shutdown  = shutdown;
 	xdrsize = 0;
+	printk(KERN_INFO "***** %s is called. *****\n", __func__);
 	while (prog) {
+		printk(KERN_INFO "***** prog=%p *****\n", prog);
 		prog->pg_lovers = prog->pg_nvers-1;
 		for (vers=0; vers<prog->pg_nvers ; vers++)
 			if (prog->pg_vers[vers]) {
@@ -389,8 +391,13 @@ __svc_create(struct svc_program *prog, u
 				if (prog->pg_vers[vers]->vs_xdrsize > xdrsize)
 					xdrsize = prog->pg_vers[vers]->vs_xdrsize;
 			}
+			else
+				printk(KERN_INFO
+				       "***** prog->pg_vers[%u]=NULL *****\n",
+				       vers);
 		prog = prog->pg_next;
 	}
+	printk(KERN_INFO "***** xdrsize=%u *****\n", xdrsize);
 	serv->sv_xdrsize   = xdrsize;
 	INIT_LIST_HEAD(&serv->sv_tempsocks);
 	INIT_LIST_HEAD(&serv->sv_permsocks);
@@ -1084,6 +1091,11 @@ svc_process_common(struct svc_rqst *rqst
 	procp->pc_count++;
 
 	/* Initialize storage for argp and resp */
+	printk(KERN_INFO "rqstp=%p procp=%p\n", rqstp, procp);
+	printk(KERN_INFO "rqstp->rq_argp=%p procp->pc_argsize=%u\n",
+	       rqstp->rq_argp, procp->pc_argsize);
+	printk(KERN_INFO "rqstp->rq_resp=%p procp->pc_ressize=%u\n",
+	       rqstp->rq_resp, procp->pc_ressize);
 	memset(rqstp->rq_argp, 0, procp->pc_argsize);
 	memset(rqstp->rq_resp, 0, procp->pc_ressize);
 

--- 2.6.35 ---

Booting.

[   27.086953] ifconfig used greatest stack depth: 1364 bytes left
[   27.255143] pcnet32 0000:02:00.0: eth0: link up
[   35.976256] mv used greatest stack depth: 1052 bytes left
[   37.993094] ***** nfsd_reset_versions is called *****.
[   37.995126] ***** found_one=0 *****.
[   37.996103] ***** &nfsd_program=c1540780 nfsd_version=c1540770 *****.
[   38.018003] ***** NFSD_MINVERS=2 NFSD_NRVERS=4 *****.
[   38.019387] ***** __svc_create is called. *****
[   38.020496] ***** prog=c1540780 *****
[   38.021391] ***** prog->pg_vers[0]=NULL *****
[   38.022425] ***** prog->pg_vers[1]=NULL *****
[   38.023470] ***** xdrsize=544 *****
[   38.069845] ***** __svc_create is called. *****
[   38.070957] ***** prog=c1541a00 *****
[   38.071844] ***** prog->pg_vers[0]=NULL *****
[   38.072883] ***** prog->pg_vers[2]=NULL *****
[   38.073941] ***** xdrsize=344 *****
[   38.149718] NET: Registered protocol family 10
[   38.588799] svc: failed to register lockdv1 RPC service (errno 97).
[   38.664394] rqstp=dc81f000 procp=c1541220
[   38.665395] rqstp->rq_argp=dcb93bf0 procp->pc_argsize=4
[   38.666621] rqstp->rq_resp=dcb94bf0 procp->pc_ressize=4
[   40.129085] ***** nfsd_reset_versions is called *****.
[   40.130336] ***** found_one=1 *****.

Doing "mount 127.0.0.1:/usr/src/ /mnt/".

[   75.786438] rqstp=de136000 procp=c1541220
[   75.787464] rqstp->rq_argp=dc81abf0 procp->pc_argsize=4
[   75.788681] rqstp->rq_resp=dc850bf0 procp->pc_ressize=4
[   75.792740] rqstp=de136000 procp=c15414cc
[   75.793701] rqstp->rq_argp=dc81abf0 procp->pc_argsize=264
[   75.815618] rqstp->rq_resp=dc850bf0 procp->pc_ressize=44
[   75.825175] rqstp=de136000 procp=c1541244
[   75.847017] rqstp->rq_argp=dc81abf0 procp->pc_argsize=264
[   75.848320] rqstp->rq_resp=dc850bf0 procp->pc_ressize=344
[   75.854935] rqstp=de136000 procp=c15414cc
[   75.855983] rqstp->rq_argp=dc81abf0 procp->pc_argsize=264
[   75.877639] rqstp->rq_resp=dc850bf0 procp->pc_ressize=44
[   75.879404] rqstp=de136000 procp=c1541244
[   75.880366] rqstp->rq_argp=dc81abf0 procp->pc_argsize=264
[   75.881639] rqstp->rq_resp=dc850bf0 procp->pc_ressize=344

--- 2.6.35-next-20100802 + 3deb279d6e5625407919a875db3a2461199566b3 ---

Booting.

[   26.414571] ifconfig used greatest stack depth: 1028 bytes left
[   26.587372] pcnet32 0000:02:00.0: eth0: link up
[   36.854504] ***** __svc_create is called. *****
[   36.861266] ***** prog=c154c760 *****
[   36.862180] ***** prog->pg_vers[0]=NULL *****
[   36.863221] ***** prog->pg_vers[1]=NULL *****
[   36.864255] ***** prog->pg_vers[2]=NULL *****
[   36.865284] ***** prog->pg_vers[3]=NULL *****
[   36.866356] ***** xdrsize=0 *****
[   36.874007] ***** __svc_create is called. *****
[   36.875094] ***** prog=c154da00 *****
[   36.875978] ***** prog->pg_vers[0]=NULL *****
[   36.877017] ***** prog->pg_vers[2]=NULL *****
[   36.878063] ***** xdrsize=344 *****
[   36.992851] NET: Registered protocol family 10
[   37.416006] svc: failed to register lockdv1 RPC service (errno 97).
[   37.419146] ***** nfsd_reset_versions is called *****.
[   37.420383] ***** found_one=0 *****.
[   37.421255] ***** &nfsd_program=c154c760 nfsd_version=c154c750 *****.
[   37.422776] ***** NFSD_MINVERS=2 NFSD_NRVERS=4 *****.

Doing "mount 127.0.0.1:/usr/src/ /mnt/".

[   58.947605] rqstp=dcfb2000 procp=c154ca20
[   58.948668] rqstp->rq_argp=00000010 procp->pc_argsize=4
[   58.949976] rqstp->rq_resp=00000010 procp->pc_ressize=4
[   58.951520] BUG: unable to handle kernel NULL pointer dereference at 00000010
[   58.953374] IP: [<c1356f20>] svc_process_common+0x370/0x640


J. Bruce Fields wrote:
> OK, I think it's another startup-order problem: depending on how things
> are started up, sv_nrthreads may already be nonzero, causing us to skip
> nfsd_reset_versions(), so that the loop in __svc_create() ends up
> leaving xdrsize 0, and then the kmalloc's in svc_prepare_thread() assign
> ZERO_SIZE_PTR.

Indeed.
Regarding 2.6.35, nfsd_reset_versions() is called before __svc_create() is
called and xdrsize != 0. But regarding 2.6.35-next-20100802 +
3deb279d6e5625407919a875db3a2461199566b3, __svc_create() is called before
nfsd_reset_versions() is called and xdrsize == 0.

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Thu, Aug 05, 2010 at 05:31:07PM -0400, J. Bruce Fields wrote:
> On Thu, Aug 05, 2010 at 04:46:12PM -0400, J. Bruce Fields wrote:
> > On Thu, Aug 05, 2010 at 10:10:16AM +0900, Tetsuo Handa wrote:
> > > J. Bruce Fields wrote:
> > > > Maybe figuring out exactly hwere that is would help work out what's
> > > > going on.  Doing
> > > > 
> > > > 	make net/sunrpc/svc.lst
> > > > 
> > > > then looking for c1356dd4 (or just mailing me svc.lst) could help.
> > > 
> > > "make net/sunrpc/svc.lst" failed due to following error.
> > > 
> > >   BFD: Dwarf Error: Abbrev offset (3238007024) greater than or equal to .debug_abbrev size (1607).
> > > 
> > > Manual printk() debug reported that
> > > rqstp->rq_argp == rqstp->rq_resp == ZERO_SIZE_PTR and
> > 
> > Huh.  As far as I can tell that will only happen if you've not no nfsd
> > versions defined; how is that happening?
> 
> OK, I think it's another startup-order problem: depending on how things
> are started up, sv_nrthreads may already be nonzero, causing us to skip
> nfsd_reset_versions(), so that the loop in __svc_create() ends up
> leaving xdrsize 0, and then the kmalloc's in svc_prepare_thread() assign
> ZERO_SIZE_PTR.
> 
> I need to think a little more about what we should be doing here.

Bah, so what you were hitting was simple--I just moved the
nfsd_reset_versions() call to the wrong place; the below should fix it.

There's also a couple other bugs in the area.

Thanks for the -next testing!

--b.

commit e844a7b9805a2b74cfd34c8604f5bba3e0869305
Author: J. Bruce Fields <bfields@redhat.com>
Date:   Fri Aug 6 15:48:03 2010 -0400

    nfsd: initialize nfsd versions before creating svc
    
    Commit 59db4a0c102e0de226a3395dbf25ea51bf845937 "nfsd: move more into
    nfsd_startup()" inadvertently moved nfsd_versions after
    nfsd_create_svc().  On older distributions using an rpc.nfsd that does
    not explicitly set the list of nfsd versions, this results in
    svc-create_pooled() being called with an empty versions array.  The
    resulting incomplete initialization leads to a NULL dereference in
    svc_process_common() the first time a client accesses the server.
    
    Move nfsd_reset_versions() back before the svc_create_pooled(); this
    time, put it closer to the svc_create_pooled() call, to make this
    mistake more difficult in the future.
    
    Signed-off-by: J. Bruce Fields <bfields@redhat.com>

diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 39ced4a..e2c4346 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -224,7 +224,6 @@ static int nfsd_startup(unsigned short port, int nrservs)
 	ret = nfs4_state_start();
 	if (ret)
 		goto out_lockd;
-	nfsd_reset_versions();
 	nfsd_up = true;
 	return 0;
 out_lockd:
@@ -329,6 +328,7 @@ int nfsd_create_serv(void)
 		       nfsd_max_blksize >= 8*1024*2)
 			nfsd_max_blksize /= 2;
 	}
+	nfsd_reset_versions();
 
 	nfsd_serv = svc_create_pooled(&nfsd_program, nfsd_max_blksize,
 				      nfsd_last_thread, nfsd, THIS_MODULE);

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Fri, Aug 06, 2010 at 05:27:28PM -0400, J. Bruce Fields wrote:
> Bah, so what you were hitting was simple--I just moved the
> nfsd_reset_versions() call to the wrong place; the below should fix it.
> 
> There's also a couple other bugs in the area.

This isn't a serious bug, but I think it makes sense to fix it.

--b.

commit 7fa53cc872332b265bc5ba1266f39586f218ad4a
Author: J. Bruce Fields <bfields@redhat.com>
Date:   Fri Aug 6 18:00:33 2010 -0400

    nfsd: don't allow setting maxblksize after svc created
    
    It's harmless to set this after the server is created, but also
    ineffective, since the value is only used at the time of
    svc_create_pooled().  So fail the attempt, in keeping with the pattern
    set by write_versions, write_{lease,grace}time and write_recoverydir.
    
    (This could break userspace that tried to write to nfsd/max_block_size
    between setting up sockets and starting the server.  However, such code
    wouldn't have worked anyway, and I don't know of any examples--rpc.nfsd
    in nfs-utils, probably the only user of the interface, doesn't do that.)
    
    Signed-off-by: J. Bruce Fields <bfields@redhat.com>

diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index 12f0ee7..b53b1d0 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -1190,7 +1190,7 @@ static ssize_t write_maxblksize(struct file *file, char *buf, size_t size)
 			bsize = NFSSVC_MAXBLKSIZE;
 		bsize &= ~(1024-1);
 		mutex_lock(&nfsd_mutex);
-		if (nfsd_serv && nfsd_serv->sv_nrthreads) {
+		if (nfsd_serv) {
 			mutex_unlock(&nfsd_mutex);
 			return -EBUSY;
 		}

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Fri, Aug 06, 2010 at 06:05:37PM -0400, J. Bruce Fields wrote:
> On Fri, Aug 06, 2010 at 05:27:28PM -0400, J. Bruce Fields wrote:
> > Bah, so what you were hitting was simple--I just moved the
> > nfsd_reset_versions() call to the wrong place; the below should fix it.
> > 
> > There's also a couple other bugs in the area.

And also there was one more problem with my original "nfsd: fix
startup/shutdown order bug": it doesn't work to use sv_nrthreads
changing from zero to nonzero as the signal for when to do all this
startup, because write_pool_threads() adjusts the number of threads
without calling nfsd_svc().  (Maybe that should be fixed.)

For now, just use the nfsd_up variable to keep track of this (which is a
little closer to Jeff's original solution).

This is a replacement.

--b.

commit 4cd7eb015e92f7cefb43eaab3e111d1b3c7b3cbf
Author: J. Bruce Fields <bfields@redhat.com>
Date:   Mon Aug 2 14:12:44 2010 -0400

    nfsd: fix startup/shutdown order bug
    
    We must create the server before we can call init_socks or check the
    number of threads.
    
    Symptoms were a NULL pointer dereference in nfsd_svc().  Problem
    identified by Jeff Layton.
    
    Also fix a minor cleanup-on-error case in nfsd_startup().
    
    Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
    Signed-off-by: J. Bruce Fields <bfields@redhat.com>

diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 92173bd..2a20f89 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -204,6 +204,9 @@ static bool nfsd_up = false;
 static int nfsd_startup(unsigned short port, int nrservs)
 {
 	int ret;
+
+	if (nfsd_up)
+		return 0;
 	/*
 	 * Readahead param cache - will no-op if it already exists.
 	 * (Note therefore results will be suboptimal if number of
@@ -217,7 +220,7 @@ static int nfsd_startup(unsigned short port, int nrservs)
 		goto out_racache;
 	ret = lockd_up();
 	if (ret)
-		return ret;
+		goto out_racache;
 	ret = nfs4_state_start();
 	if (ret)
 		goto out_lockd;
@@ -420,7 +423,7 @@ int
 nfsd_svc(unsigned short port, int nrservs)
 {
 	int	error;
-	bool	first_thread;
+	bool	nfsd_up_before;
 
 	mutex_lock(&nfsd_mutex);
 	dprintk("nfsd: creating service\n");
@@ -432,29 +435,29 @@ nfsd_svc(unsigned short port, int nrservs)
 	if (nrservs == 0 && nfsd_serv == NULL)
 		goto out;
 
-	first_thread = (nfsd_serv->sv_nrthreads == 0) && (nrservs != 0);
-
-	if (first_thread) {
-		error = nfsd_startup(port, nrservs);
-		if (error)
-			goto out;
-	}
 	error = nfsd_create_serv();
 	if (error)
-		goto out_shutdown;
-	error = svc_set_num_threads(nfsd_serv, NULL, nrservs);
+		goto out;
+
+	nfsd_up_before = nfsd_up;
+
+	error = nfsd_startup(port, nrservs);
 	if (error)
 		goto out_destroy;
+	}
+	error = svc_set_num_threads(nfsd_serv, NULL, nrservs);
+	if (error)
+		goto out_shutdown;
 	/* We are holding a reference to nfsd_serv which
 	 * we don't want to count in the return value,
 	 * so subtract 1
 	 */
 	error = nfsd_serv->sv_nrthreads - 1;
-out_destroy:
-	svc_destroy(nfsd_serv);		/* Release server */
 out_shutdown:
-	if (error < 0 && first_thread)
+	if (error < 0 && !nfsd_up_before)
 		nfsd_shutdown();
+out_destroy:
+	svc_destroy(nfsd_serv);		/* Release server */
 out:
 	mutex_unlock(&nfsd_mutex);
 	return error;

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[Tetsuo Handa]

Applying commit 4cd7eb015e92f7cefb43eaab3e111d1b3c7b3cbf (with below patch)
and commit e844a7b9805a2b74cfd34c8604f5bba3e0869305 and
commit 7fa53cc872332b265bc5ba1266f39586f218ad4a on linux-2.6.35-next-20100802
solved all problems found in my environment.

Thank you.
--------------------

Fix build error by commit 4cd7eb015e92f7cefb43eaab3e111d1b3c7b3cbf
"nfsd: fix startup/shutdown order bug".

---
 fs/nfsd/nfssvc.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- linux-2.6.35-next.orig/fs/nfsd/nfssvc.c
+++ linux-2.6.35-next/fs/nfsd/nfssvc.c
@@ -444,7 +444,7 @@ nfsd_svc(unsigned short port, int nrserv
 	error = nfsd_startup(port, nrservs);
 	if (error)
 		goto out_destroy;
-	}
+
 	error = svc_set_num_threads(nfsd_serv, NULL, nrservs);
 	if (error)
 		goto out_shutdown;

Re: linux-next NFSD: NULL pointer dereference at nfsd_svc()

[J. Bruce Fields]

On Sat, Aug 07, 2010 at 10:48:02AM +0900, Tetsuo Handa wrote:
> Applying commit 4cd7eb015e92f7cefb43eaab3e111d1b3c7b3cbf (with below patch)
> and commit e844a7b9805a2b74cfd34c8604f5bba3e0869305 and
> commit 7fa53cc872332b265bc5ba1266f39586f218ad4a on linux-2.6.35-next-20100802
> solved all problems found in my environment.
> 
> Thank you.

Thank you for the confirmation.

> --------------------
> 
> Fix build error by commit 4cd7eb015e92f7cefb43eaab3e111d1b3c7b3cbf
> "nfsd: fix startup/shutdown order bug".

Yes, apologies for that--I was compiling as sent that out!

--b.

> 
> ---
>  fs/nfsd/nfssvc.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> --- linux-2.6.35-next.orig/fs/nfsd/nfssvc.c
> +++ linux-2.6.35-next/fs/nfsd/nfssvc.c
> @@ -444,7 +444,7 @@ nfsd_svc(unsigned short port, int nrserv
>  	error = nfsd_startup(port, nrservs);
>  	if (error)
>  		goto out_destroy;
> -	}
> +
>  	error = svc_set_num_threads(nfsd_serv, NULL, nrservs);
>  	if (error)
>  		goto out_shutdown;