nfs-utils: rpc.svcgssd segmentation fault in nss_gss_princ_to_ids()

nfs-utils: rpc.svcgssd segmentation fault in nss_gss_princ_to_ids()

[Matthias Gerstner]

[-- Attachment #1.1: Type: text/plain, Size: 4162 bytes --]

Hello!

I hope I have found the right place to report this.

I have recently upgraded from nfs-utils-1.3.4 to nfs-utils-2.3.1 on a Gentoo
Linux (hardened) system. After this upgrade the rpc.svcgssd crashes on my
kerberized NFS server each time a client tries to mount an NFS export.

The crash is a segmentation fault:

  kernel: rpc.svcgssd[19772]: segfault at 8 ip 00007f6b6aafa98c sp 00007fff2ca8ce30 error 4 in nsswitch.so[7f6b6aaf9000+7000]

The call stack leading up to this looks as follows:

  (gdb) 
  Program received signal SIGSEGV, Segmentation fault.
  nss_gss_princ_to_ids (secname=<optimized out>, princ=0x55cabc7d9940 "nfs/host.example.domain@EXAMPLE.HOME", 
      uid=0x7ffca550e0d8, gid=0x7ffca550e0dc, UNUSED_ex=<optimized out>) at nss.c:415
  415	nss.c: No such file or directory.

  (gdb) bt
  #0  nss_gss_princ_to_ids (secname=<optimized out>, princ=0x55cabc7d9940 "nfs/host.example.domain@EXAMPLE.HOME", 
      uid=0x7ffca550e0d8, gid=0x7ffca550e0dc, UNUSED_ex=<optimized out>) at nss.c:415
  #1  0x00007f6c97cc2886 in nfs4_gss_princ_to_ids (secname=secname@entry=0x55caba8e2030 <m2f+16> "krb5", 
      princ=princ@entry=0x55cabc7d9940 "nfs/host.example.domain@EXAMPLE.HOME", uid=0x7ffca550e0d8, uid@entry=0x1, 
      gid=0x7ffca550e0dc, gid@entry=0x7f6c95cf3a76 <conf_get_section+86>) at libnfsidmap.c:682
  #2  0x000055caba6da138 in get_ids (cred=0x7ffca550e038, mech=<optimized out>, client_name=<optimized out>)
      at svcgssd_proc.c:251
  #3  handle_nullreq (f=f@entry=3) at svcgssd_proc.c:407
  #4  0x000055caba6d996a in gssd_run () at svcgssd_main_loop.c:91
  #5  0x000055caba6d83b1 in main (argc=1, argv=<optimized out>) at svcgssd.c:211

  (gdb) p realms
  $1 = (struct conf_list *) 0x0

I debugged this a bit and strange things came about. There seem to be
duplicate instances of the local_realms variable in the
support/nfsidmap/nfsidmap_common.c compilation unit:

  gdb --args /usr/sbin/rpc.svcgssd  -f
  (gdb) start
  Temporary breakpoint 1 at 0x2170: file svcgssd.c, line 94.
  Starting program: /usr/sbin/rpc.svcgssd -f
  Temporary breakpoint 1, main (argc=2, argv=0x7fffffffddc8) at svcgssd.c:94
  94	svcgssd.c: No such file or directory.

  (gdb) b get_local_realms 
  Breakpoint 2 at 0x7ffff7bd1fc0: file nfsidmap_common.c, line 30.
  (gdb) c
  Continuing.

  Breakpoint 2, get_local_realms () at nfsidmap_common.c:30
  30	nfsidmap_common.c: No such file or directory.
  (gdb) disassemble 
  Dump of assembler code for function get_local_realms:
  => 0x00007ffff5e0b4a0 <+0>:	sub    $0x1038,%rsp
     0x00007ffff5e0b4a7 <+7>:	orq    $0x0,(%rsp)
     0x00007ffff5e0b4ac <+12>:	add    $0x1020,%rsp
     0x00007ffff5e0b4b3 <+19>:	mov    %fs:0x28,%rax
     0x00007ffff5e0b4bc <+28>:	mov    %rax,0x8(%rsp)
     0x00007ffff5e0b4c1 <+33>:	xor    %eax,%eax
     0x00007ffff5e0b4c3 <+35>:	mov    0x204f7e(%rip),%rax        # 0x7ffff6010448 <local_realms>
     0x00007ffff5e0b4ca <+42>:	mov    0x8(%rsp),%rdx
     0x00007ffff5e0b4cf <+47>:	xor    %fs:0x28,%rdx
     0x00007ffff5e0b4d8 <+56>:	jne    0x7ffff5e0b4df <get_local_realms+63>
     0x00007ffff5e0b4da <+58>:	add    $0x18,%rsp
     0x00007ffff5e0b4de <+62>:	retq   
     0x00007ffff5e0b4df <+63>:	callq  0x7ffff5e0a250 <__stack_chk_fail@plt>

  End of assembler dump.

  (gdb) p local_realms
  $1 = (struct conf_list *) 0x555555764300
  (gdb) p &local_realms
  $2 = (struct conf_list **) 0x7ffff7dd7208 <local_realms>

  (gdb) x /1g 0x7ffff6010448
  0x7ffff6010448 <local_realms>:	0x0000000000000000

So there's one instance of the variable that was actually assigned the
expected data. And there's another one that is at NULL and returned from
get_local_realms().

This is related to the "#pragma GCC visibility push(hidden)" in the
compilation unit. Moving it after the local_realms variable declaration fixes
the issue. I've attached a patch that does just this.

I am not quite sure whether this is a compiler issue or an invalid use of the
visibility hidden pragma. The compiler used is:

  gcc (Gentoo Hardened 6.4.0-r1 p1.3) 6.4.0

Please advise.

Regards

Matthias

[-- Attachment #1.2: local_realms_segfault.patch --]
[-- Type: text/x-diff, Size: 1109 bytes --]

commit b1935a982aa68890ffe888381f7905f57c6b055d
Author: Matthias Gerstner <Matthias.Gerstner@nefkom.net>
Date:   Tue Feb 20 23:33:18 2018 +0100

    get_local_realms: work around strange duplication of local_realms
    
    The hidden visibility seems to cause the get_local_realms() function to
    return a NULL pointer, although the local_realms global variable is
    actually assigned. Assembler code shows there are two instances of this
    variable around, one set to NULL, the other to the expected value.
    
    This happened on Gentoo Hardened with gcc 4.6.0-r1.

diff --git a/support/nfsidmap/nfsidmap_common.c b/support/nfsidmap/nfsidmap_common.c
index 891c855..e5c33a8 100644
--- a/support/nfsidmap/nfsidmap_common.c
+++ b/support/nfsidmap/nfsidmap_common.c
@@ -19,13 +19,13 @@
 #include "nfsidmap_plugin.h"
 #include "conffile.h"
 
-#pragma GCC visibility push(hidden)
-
 int reformat_group = 0;
 int no_strip = 0;
 
 struct conf_list *local_realms;
 
+#pragma GCC visibility push(hidden)
+
 struct conf_list *get_local_realms(void)
 {
 	return local_realms;

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: nfs-utils: rpc.svcgssd segmentation fault in nss_gss_princ_to_ids()

[Steve Dickson]

Hello,

On 02/20/2018 05:48 PM, Matthias Gerstner wrote:
> Hello!
> 
> I hope I have found the right place to report this.
It is... But in the future please in line your
patch (which I have a the bottom of this reply)
instead attaching it.. I just makes it easier 
to review.

> 
> I have recently upgraded from nfs-utils-1.3.4 to nfs-utils-2.3.1 on a Gentoo
> Linux (hardened) system. After this upgrade the rpc.svcgssd crashes on my
> kerberized NFS server each time a client tries to mount an NFS export.
The main difference in these releases is we rolled libnfsidmap into
nfs-utils... 

Also note, at least with Fedora, we no longer use rpc.svcgssd
instead we use gssproxy to access GSSAPI credentials

> 
> The crash is a segmentation fault:
> 
>   kernel: rpc.svcgssd[19772]: segfault at 8 ip 00007f6b6aafa98c sp 00007fff2ca8ce30 error 4 in nsswitch.so[7f6b6aaf9000+7000]
> 
> The call stack leading up to this looks as follows:
> 
>   (gdb) 
>   Program received signal SIGSEGV, Segmentation fault.
>   nss_gss_princ_to_ids (secname=<optimized out>, princ=0x55cabc7d9940 "nfs/host.example.domain@EXAMPLE.HOME", 
>       uid=0x7ffca550e0d8, gid=0x7ffca550e0dc, UNUSED_ex=<optimized out>) at nss.c:415
>   415	nss.c: No such file or directory.
> 
>   (gdb) bt
>   #0  nss_gss_princ_to_ids (secname=<optimized out>, princ=0x55cabc7d9940 "nfs/host.example.domain@EXAMPLE.HOME", 
>       uid=0x7ffca550e0d8, gid=0x7ffca550e0dc, UNUSED_ex=<optimized out>) at nss.c:415
>   #1  0x00007f6c97cc2886 in nfs4_gss_princ_to_ids (secname=secname@entry=0x55caba8e2030 <m2f+16> "krb5", 
>       princ=princ@entry=0x55cabc7d9940 "nfs/host.example.domain@EXAMPLE.HOME", uid=0x7ffca550e0d8, uid@entry=0x1, 
>       gid=0x7ffca550e0dc, gid@entry=0x7f6c95cf3a76 <conf_get_section+86>) at libnfsidmap.c:682
>   #2  0x000055caba6da138 in get_ids (cred=0x7ffca550e038, mech=<optimized out>, client_name=<optimized out>)
>       at svcgssd_proc.c:251
>   #3  handle_nullreq (f=f@entry=3) at svcgssd_proc.c:407
>   #4  0x000055caba6d996a in gssd_run () at svcgssd_main_loop.c:91
>   #5  0x000055caba6d83b1 in main (argc=1, argv=<optimized out>) at svcgssd.c:211
> 
>   (gdb) p realms
>   $1 = (struct conf_list *) 0x0
> 
> I debugged this a bit and strange things came about. There seem to be
> duplicate instances of the local_realms variable in the
> support/nfsidmap/nfsidmap_common.c compilation unit:
> 
>   gdb --args /usr/sbin/rpc.svcgssd  -f
>   (gdb) start
>   Temporary breakpoint 1 at 0x2170: file svcgssd.c, line 94.
>   Starting program: /usr/sbin/rpc.svcgssd -f
>   Temporary breakpoint 1, main (argc=2, argv=0x7fffffffddc8) at svcgssd.c:94
>   94	svcgssd.c: No such file or directory.
> 
>   (gdb) b get_local_realms 
>   Breakpoint 2 at 0x7ffff7bd1fc0: file nfsidmap_common.c, line 30.
>   (gdb) c
>   Continuing.
> 
>   Breakpoint 2, get_local_realms () at nfsidmap_common.c:30
>   30	nfsidmap_common.c: No such file or directory.
>   (gdb) disassemble 
>   Dump of assembler code for function get_local_realms:
>   => 0x00007ffff5e0b4a0 <+0>:	sub    $0x1038,%rsp
>      0x00007ffff5e0b4a7 <+7>:	orq    $0x0,(%rsp)
>      0x00007ffff5e0b4ac <+12>:	add    $0x1020,%rsp
>      0x00007ffff5e0b4b3 <+19>:	mov    %fs:0x28,%rax
>      0x00007ffff5e0b4bc <+28>:	mov    %rax,0x8(%rsp)
>      0x00007ffff5e0b4c1 <+33>:	xor    %eax,%eax
>      0x00007ffff5e0b4c3 <+35>:	mov    0x204f7e(%rip),%rax        # 0x7ffff6010448 <local_realms>
>      0x00007ffff5e0b4ca <+42>:	mov    0x8(%rsp),%rdx
>      0x00007ffff5e0b4cf <+47>:	xor    %fs:0x28,%rdx
>      0x00007ffff5e0b4d8 <+56>:	jne    0x7ffff5e0b4df <get_local_realms+63>
>      0x00007ffff5e0b4da <+58>:	add    $0x18,%rsp
>      0x00007ffff5e0b4de <+62>:	retq   
>      0x00007ffff5e0b4df <+63>:	callq  0x7ffff5e0a250 <__stack_chk_fail@plt>
> 
>   End of assembler dump.
> 
>   (gdb) p local_realms
>   $1 = (struct conf_list *) 0x555555764300
>   (gdb) p &local_realms
>   $2 = (struct conf_list **) 0x7ffff7dd7208 <local_realms>
> 
>   (gdb) x /1g 0x7ffff6010448
>   0x7ffff6010448 <local_realms>:	0x0000000000000000
> 
> So there's one instance of the variable that was actually assigned the
> expected data. And there's another one that is at NULL and returned from
> get_local_realms().
> 
> This is related to the "#pragma GCC visibility push(hidden)" in the
> compilation unit. Moving it after the local_realms variable declaration fixes
> the issue. I've attached a patch that does just this.
Nice work... BTW... 

> 
> I am not quite sure whether this is a compiler issue or an invalid use of the
> visibility hidden pragma. The compiler used is:
> 
>   gcc (Gentoo Hardened 6.4.0-r1 p1.3) 6.4.0
> 
> Please advise.
> 
> Regards
> 
> Matthias
> commit b1935a982aa68890ffe888381f7905f57c6b055d
> Author: Matthias Gerstner <Matthias.Gerstner@nefkom.net>
> Date:   Tue Feb 20 23:33:18 2018 +0100
>
>    get_local_realms: work around strange duplication of local_realms
>    
>    The hidden visibility seems to cause the get_local_realms() function to
>    return a NULL pointer, although the local_realms global variable is
>    actually assigned. Assembler code shows there are two instances of this
>    variable around, one set to NULL, the other to the expected value.
>    
>    This happened on Gentoo Hardened with gcc 4.6.0-r1.
>
> diff --git a/support/nfsidmap/nfsidmap_common.c b/support/nfsidmap/nfsidmap_common.c
> index 891c855..e5c33a8 100644
> --- a/support/nfsidmap/nfsidmap_common.c
> +++ b/support/nfsidmap/nfsidmap_common.c
> @@ -19,13 +19,13 @@
> #include "nfsidmap_plugin.h"
> #include "conffile.h"
> 
> -#pragma GCC visibility push(hidden)
> -
>  int reformat_group = 0;
>  int no_strip = 0;
> 
>  struct conf_list *local_realms;
>  
> +#pragma GCC visibility push(hidden)
By moving this pragama statement both the no_strip 
and reformat_group are also being exposed as 
global variables in the libnfsidmap, correct?
I'm not sure we want to do that, if that is the case.

I guess exposing local_realms is necessary
but it is adding to the API... which is probably
not good...

steved. 
> +
>  struct conf_list *get_local_realms(void)
>  {
 	return local_realms;

Re: nfs-utils: rpc.svcgssd segmentation fault in nss_gss_princ_to_ids()

[Justin Mitchell]

On Tue, 2018-02-20 at 23:48 +0100, Matthias Gerstner wrote:
> Hello!
> 
> I hope I have found the right place to report this.
> 
> I have recently upgraded from nfs-utils-1.3.4 to nfs-utils-2.3.1 on a Gentoo
> Linux (hardened) system. After this upgrade the rpc.svcgssd crashes on my
> kerberized NFS server each time a client tries to mount an NFS export.
<snip>
>   
> I debugged this a bit and strange things came about. There seem to be
> duplicate instances of the local_realms variable in the
> support/nfsidmap/nfsidmap_common.c compilation unit:
<snip>
> So there's one instance of the variable that was actually assigned the
> expected data. And there's another one that is at NULL and returned from
> get_local_realms().
> 
> This is related to the "#pragma GCC visibility push(hidden)" in the
> compilation unit. Moving it after the local_realms variable declaration fixes
> the issue. I've attached a patch that does just this.
> 
> I am not quite sure whether this is a compiler issue or an invalid use of the
> visibility hidden pragma. The compiler used is:

This is part of my previous work to fold libnfsidmap into the nfs-utils
tree, in this case the #pragma is used to make the local_realms global
accessible only by other parts of the library that need it, external
users such as the nss plugin get access to it via the get_local_realms()
function.

I will run some tests to see if this is isolated to certain compiler
versions, or if the pragma isnt working as I expected.

However simply moving the #pragma hidden will change the ABI which is a
problem for some, so if necessary I will instead reorganise the code to
achieve the same functionality without using the pragma

Re: nfs-utils: rpc.svcgssd segmentation fault in nss_gss_princ_to_ids()

[Matthias Gerstner]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

Hello,

thanks for your reply.

> It is... But in the future please in line your
> patch (which I have a the bottom of this reply)
> instead attaching it.. I just makes it easier 
> to review.

I will do so.

> Also note, at least with Fedora, we no longer use rpc.svcgssd
> instead we use gssproxy to access GSSAPI credentials

Okay. I have been running this NFS setup for years without touching much.
It looks like I should invest some time to check if everything is still up to
date.

> By moving this pragama statement both the no_strip 
> and reformat_group are also being exposed as 
> global variables in the libnfsidmap, correct?
> I'm not sure we want to do that, if that is the case.

I was not suggesting to accept this as a patch. I just wanted to make clear
what "helped" in my case.

I will try some more experiments when I have time. Maybe I can find a suitable
patch. Getter/Setter functions instead of the extern declared variables might
help.

Regards

Matthias

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: nfs-utils: rpc.svcgssd segmentation fault in nss_gss_princ_to_ids()

[Matthias Gerstner]

[-- Attachment #1: Type: text/plain, Size: 488 bytes --]

Hello,

> I will run some tests to see if this is isolated to certain compiler
> versions, or if the pragma isnt working as I expected.
> 
> However simply moving the #pragma hidden will change the ABI which is a
> problem for some, so if necessary I will instead reorganise the code to
> achieve the same functionality without using the pragma

yes I understand. I was not suggesting this patch to be an actual solution.

Thanks for looking into this.

Regards

Matthias

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: nfs-utils: rpc.svcgssd segmentation fault in nss_gss_princ_to_ids()

[matthias.gerstner]

[-- Attachment #1: Type: text/plain, Size: 255 bytes --]

Hello,

> I will run some tests to see if this is isolated to certain compiler
> versions, or if the pragma isnt working as I expected.

I just want to report that I just tested with a more recent gcc 7.3.0 and the
result was the same.

Regards

Matthias

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]