nfs sec=krb5 on RHEL and CentOS

nfs sec=krb5 on RHEL and CentOS

[james bardin]

I couldn't get sec=krb5 to work on clients running RHEL4 (and CentOS),
so I built a slightly newer version of nfs-utils. (nfs-utils-1.0.7-13)

I can now get the exports to mount, but I'm seeing some very strange behavior.

A mkdir return a file exists error, but the directory is created.
Redirecting std out to a file returns an ioerror, and the file is
created but empty.
Vim creates swap files, they fail so it doesn't use them, but reports
them as existing.

I can chmod, chown existing files as expected.

The server is a netapp filer. Other distros (Ubuntu, Fedora) work as
expected. FC4 with the above version of nfs-utils works filne.

thanks
-jim

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[Steve Dickson]

Sorry for the delayed response...

james bardin wrote:
> I couldn't get sec=krb5 to work on clients running RHEL4 (and CentOS),
> so I built a slightly newer version of nfs-utils. (nfs-utils-1.0.7-13)
Go back and grab the latest rhel4 nfs-utils which is 1.0.6-77. That has
been updated with a number of patches that are in the upstream
version...

> 
> A mkdir return a file exists error, but the directory is created.
> Redirecting std out to a file returns an ioerror, and the file is
> created but empty.
> Vim creates swap files, they fail so it doesn't use them, but reports
> them as existing.
> 
> I can chmod, chown existing files as expected.
hmm... if this continues after you updated your nfs-utils, please
open a bug reported and post a bzip2 binary tethereal trace
something similar to:
     tethereal -w /tmp/data.pcap host <server> ; bzip2 /tmp/data.pcap

steved.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[James Bardin]

Steve Dickson wrote:
>
> Sorry for the delayed response...
That's ok, thanks for the assistance. This is kind of a show stopper for 
us, as we have to be standardized on RHEL/CentOS this summer.
I does work with a new kernel (2.6.19.2), but that's not feasible for 
our entire install base right now.

> Go back and grab the latest rhel4 nfs-utils which is 1.0.6-77. That has
> been updated with a number of patches that are in the upstream
> version...
>
Can you point me to the package? I can't seem to find anything but 
nfs-utils-1.0.6-70.EL4.src.rpm (from redhat's ftp site), and this is the 
version I started with.

thanks
-jim



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[James Bardin]

Sorry Steve,

I accidentally rebooted into kernel 2.6.19 when I tested the patches.
Same problem still, and on our 64bit systems, everything fails with 
statfs error 13.

Here's the dmesg errors:
call_verify: auth check failed
call_verify: auth check failed
call_verify: auth check failed
RPC call_verify: retry failed, exit EIO

and:
nfs_statfs: statfs error = 13

Attached is the trace

the commands were:
$ mkdir newdir
$ ls
$ cd newdir
$ echo TESTSTRING > newfile
$ ls

thanks
-jim


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[J. Bruce Fields]

On Wed, Jan 24, 2007 at 03:58:04PM -0500, James Bardin wrote:
> Sorry Steve,
> 
> I accidentally rebooted into kernel 2.6.19 when I tested the patches.
> Same problem still, and on our 64bit systems, everything fails with 
> statfs error 13.

Just to make sure I understand--the statfs erro on 64bit systems is a
separate problem from the one described below?

--b.

> 
> Here's the dmesg errors:
> call_verify: auth check failed
> call_verify: auth check failed
> call_verify: auth check failed
> RPC call_verify: retry failed, exit EIO
> 
> and:
> nfs_statfs: statfs error = 13
> 
> Attached is the trace
> 
> the commands were:
> $ mkdir newdir
> $ ls
> $ cd newdir
> $ echo TESTSTRING > newfile
> $ ls
> 
> thanks
> -jim
> 
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[james bardin]

On 1/24/07, J. Bruce Fields <bfields@fieldses.org> wrote:
> On Wed, Jan 24, 2007 at 03:58:04PM -0500, James Bardin wrote:
> > Sorry Steve,
> >
> > I accidentally rebooted into kernel 2.6.19 when I tested the patches.
> > Same problem still, and on our 64bit systems, everything fails with
> > statfs error 13.
>
> Just to make sure I understand--the statfs erro on 64bit systems is a
> separate problem from the one described below?
>

statfs error was happening everywhere, but the 64bit systems would fail worse.

> --b.
>
> >
> > Here's the dmesg errors:
> > call_verify: auth check failed
> > call_verify: auth check failed
> > call_verify: auth check failed
> > RPC call_verify: retry failed, exit EIO
> >
> > and:
> > nfs_statfs: statfs error = 13
> >


> Try using the -o noacl mount option...
>
> steved.
>

Steved,
Ok, this seems to be it. ( I'll try out some other configuration/64bit tomorrow)
Could you give me a brief summary on why this is happening, and only
on this kernel? Mostly, should I always be using noacl when the netapp
is handling file access?

thanks
-jim

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[James Bardin]

I'm almost there!
Between the nfs-utils patch, and the noacl option, I have my 32bit 
systems working. (thanks Steve)

On x86_64, I'm having kerberos problems (exact same config):

rpc.gssd[4871]: handling krb5 upcall
rpc.gssd[4871]: getting credentials for client with uid xxxx for server 
yyyy.bu.edu
rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' being considered
rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' matches name check and has 
mtime of 1169750861
rpc.gssd[4871]: using FILE:/tmp/krb5cc_xxxx_bSULEy as credentials cache 
for client with uid xxxx for server yyyy.bu.edu
rpc.gssd[4871]: creating context using euid xxxx (save_uid 0)
rpc.gssd[4871]: creating tcp client for server yyyy.bu.edu
rpc.gssd[4871]: WARNING: can't create rpc_clnt for server engna1.bu.edu 
for user with uid xxxx: RPC: Success 
rpc.gssd[4871]: WARNING: Failed to create krb5 context for user with uid 
xxxx for server yyyy.bu.edu
rpc.gssd[4871]: doing error downcall


-jim

>> Try using the -o noacl mount option...
>>
>> steved.
>>
>
> Steved,
> Ok, this seems to be it. ( I'll try out some other configuration/64bit 
> tomorrow)
> Could you give me a brief summary on why this is happening, and only
> on this kernel? Mostly, should I always be using noacl when the netapp
> is handling file access?
>
> thanks
> -jim
>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[James Bardin]

> I'm almost there!
> Between the nfs-utils patch, and the noacl option, I have my 32bit 
> systems working. (thanks Steve)
>
> On x86_64, I'm having kerberos problems (exact same config):
>
> rpc.gssd[4871]: handling krb5 upcall
> rpc.gssd[4871]: getting credentials for client with uid xxxx for 
> server yyyy.bu.edu
> rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' being considered
> rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' matches name check and 
> has mtime of 1169750861
> rpc.gssd[4871]: using FILE:/tmp/krb5cc_xxxx_bSULEy as credentials 
> cache for client with uid xxxx for server yyyy.bu.edu
> rpc.gssd[4871]: creating context using euid xxxx (save_uid 0)
> rpc.gssd[4871]: creating tcp client for server yyyy.bu.edu
> rpc.gssd[4871]: WARNING: can't create rpc_clnt for server 
> engna1.bu.edu for user with uid xxxx: RPC: Success rpc.gssd[4871]: 
> WARNING: Failed to create krb5 context for user with uid xxxx for 
> server yyyy.bu.edu
> rpc.gssd[4871]: doing error downcall
>
>
x86_64 is working on an older version, I read the errata, and it 
shouldn't effect us, but something is wrong in the new ones. This is 
with sec=krb5.
nfs-utils-1.0.6-77 causes the above problems
nfs-utils-1.0.6-70 will hang on rpc.gssd
nfs-utils-1.0.6-65 is working.



> Kevin Coffman wrote:
> Unless Steve has pulled in more fixes than I think, you'll probably
> need to upgrade libgssapi and maybe nfs-utils to get 64-bit working.
> I was going to look at what is in nfs-utils-1.0.6-77, but my RHEL 4
> subscription has expired :-/.  Working on that.
>
> Offhand, I don't recall exactly when those changes went in, but I'll 
> check. 
I'm trying to keep to the RHEL src tree as much as possible, this is 
going on a lot of machines.
Do you know of a patch/update for libgssapi that I could try?

thanks
-jim



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[Kevin Coffman]

On 1/25/07, James Bardin <jbardin@bu.edu> wrote:
>
> > I'm almost there!
> > Between the nfs-utils patch, and the noacl option, I have my 32bit
> > systems working. (thanks Steve)
> >
> > On x86_64, I'm having kerberos problems (exact same config):
> >
> > rpc.gssd[4871]: handling krb5 upcall
> > rpc.gssd[4871]: getting credentials for client with uid xxxx for
> > server yyyy.bu.edu
> > rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' being considered
> > rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' matches name check and
> > has mtime of 1169750861
> > rpc.gssd[4871]: using FILE:/tmp/krb5cc_xxxx_bSULEy as credentials
> > cache for client with uid xxxx for server yyyy.bu.edu
> > rpc.gssd[4871]: creating context using euid xxxx (save_uid 0)
> > rpc.gssd[4871]: creating tcp client for server yyyy.bu.edu
> > rpc.gssd[4871]: WARNING: can't create rpc_clnt for server
> > engna1.bu.edu for user with uid xxxx: RPC: Success rpc.gssd[4871]:
> > WARNING: Failed to create krb5 context for user with uid xxxx for
> > server yyyy.bu.edu
> > rpc.gssd[4871]: doing error downcall
> >
> >
> x86_64 is working on an older version, I read the errata, and it
> shouldn't effect us, but something is wrong in the new ones. This is
> with sec=krb5.
> nfs-utils-1.0.6-77 causes the above problems
> nfs-utils-1.0.6-70 will hang on rpc.gssd
> nfs-utils-1.0.6-65 is working.
>
>
>
> > Kevin Coffman wrote:
> > Unless Steve has pulled in more fixes than I think, you'll probably
> > need to upgrade libgssapi and maybe nfs-utils to get 64-bit working.
> > I was going to look at what is in nfs-utils-1.0.6-77, but my RHEL 4
> > subscription has expired :-/.  Working on that.
> >
> > Offhand, I don't recall exactly when those changes went in, but I'll
> > check.
> I'm trying to keep to the RHEL src tree as much as possible, this is
> going on a lot of machines.
> Do you know of a patch/update for libgssapi that I could try?
>
> thanks
> -jim

OK, sorry for the false alarm.  I've looked closer at what is included
in nfs-utils-1.0.6-70 on another RHEL system here.  From what I can
tell, the -70 version should have the 64-bit fixes that I am aware of.
 It sounds like the -65 had them as well.

It looks like Steve Dickson has been working hard!  He may be the best
person to say what changed between nfs-utils-1.0.6-65 and -70 and -77
versions that might be causing your problems.

K.C.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[James Bardin]

Kevin Coffman wrote:
> On 1/25/07, James Bardin <jbardin@bu.edu> wrote:
>>
>> > I'm almost there!
>> > Between the nfs-utils patch, and the noacl option, I have my 32bit
>> > systems working. (thanks Steve)
>> >
>> > On x86_64, I'm having kerberos problems (exact same config):
>> >
>> > rpc.gssd[4871]: handling krb5 upcall
>> > rpc.gssd[4871]: getting credentials for client with uid xxxx for
>> > server yyyy.bu.edu
>> > rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' being considered
>> > rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' matches name check and
>> > has mtime of 1169750861
>> > rpc.gssd[4871]: using FILE:/tmp/krb5cc_xxxx_bSULEy as credentials
>> > cache for client with uid xxxx for server yyyy.bu.edu
>> > rpc.gssd[4871]: creating context using euid xxxx (save_uid 0)
>> > rpc.gssd[4871]: creating tcp client for server yyyy.bu.edu
>> > rpc.gssd[4871]: WARNING: can't create rpc_clnt for server
>> > engna1.bu.edu for user with uid xxxx: RPC: Success rpc.gssd[4871]:
>> > WARNING: Failed to create krb5 context for user with uid xxxx for
>> > server yyyy.bu.edu
>> > rpc.gssd[4871]: doing error downcall
>> >
>> >
>> x86_64 is working on an older version, I read the errata, and it
>> shouldn't effect us, but something is wrong in the new ones. This is
>> with sec=krb5.
>> nfs-utils-1.0.6-77 causes the above problems
>> nfs-utils-1.0.6-70 will hang on rpc.gssd
>> nfs-utils-1.0.6-65 is working.
>>
>>
>>
>> > Kevin Coffman wrote:
>> > Unless Steve has pulled in more fixes than I think, you'll probably
>> > need to upgrade libgssapi and maybe nfs-utils to get 64-bit working.
>> > I was going to look at what is in nfs-utils-1.0.6-77, but my RHEL 4
>> > subscription has expired :-/.  Working on that.
>> >
>> > Offhand, I don't recall exactly when those changes went in, but I'll
>> > check.
>> I'm trying to keep to the RHEL src tree as much as possible, this is
>> going on a lot of machines.
>> Do you know of a patch/update for libgssapi that I could try?
>>
>> thanks
>> -jim
>
> OK, sorry for the false alarm.  I've looked closer at what is included
> in nfs-utils-1.0.6-70 on another RHEL system here.  From what I can
> tell, the -70 version should have the 64-bit fixes that I am aware of.
> It sounds like the -65 had them as well.
>
> It looks like Steve Dickson has been working hard!  He may be the best
> person to say what changed between nfs-utils-1.0.6-65 and -70 and -77
> versions that might be causing your problems.
>
> K.C.
>
I don't know if it's related, but sometimes when I build an nfs-utils 
src.rpm, it dumps out saying the GSS with KRB5 support not found. If I 
try to build again, it works???




-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: nfs sec=krb5 on RHEL and CentOS

[James Bardin]

>
>> On 1/25/07, James Bardin <jbardin@bu.edu> wrote:
>>>
>>> > I'm almost there!
>>> > Between the nfs-utils patch, and the noacl option, I have my 32bit
>>> > systems working. (thanks Steve)
>>> >
>>> > On x86_64, I'm having kerberos problems (exact same config):
>>> >
>>> > rpc.gssd[4871]: handling krb5 upcall
>>> > rpc.gssd[4871]: getting credentials for client with uid xxxx for
>>> > server yyyy.bu.edu
>>> > rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' being considered
>>> > rpc.gssd[4871]: CC file 'krb5cc_xxxx_bSULEy' matches name check and
>>> > has mtime of 1169750861
>>> > rpc.gssd[4871]: using FILE:/tmp/krb5cc_xxxx_bSULEy as credentials
>>> > cache for client with uid xxxx for server yyyy.bu.edu
>>> > rpc.gssd[4871]: creating context using euid xxxx (save_uid 0)
>>> > rpc.gssd[4871]: creating tcp client for server yyyy.bu.edu
>>> > rpc.gssd[4871]: WARNING: can't create rpc_clnt for server
>>> > engna1.bu.edu for user with uid xxxx: RPC: Success rpc.gssd[4871]:
>>> > WARNING: Failed to create krb5 context for user with uid xxxx for
>>> > server yyyy.bu.edu
>>> > rpc.gssd[4871]: doing error downcall
>>> >
>>> >
>>> x86_64 is working on an older version, I read the errata, and it
>>> shouldn't effect us, but something is wrong in the new ones. This is
>>> with sec=krb5.
>>> nfs-utils-1.0.6-77 causes the above problems
>>> nfs-utils-1.0.6-70 will hang on rpc.gssd
>>> nfs-utils-1.0.6-65 is working.
>>>
>>
> I don't know if it's related, but sometimes when I build an nfs-utils 
> src.rpm, it dumps out saying the GSS with KRB5 support not found. If I 
> try to build again, it works???
>


I've been testing on CentOS so far with the above results. 
Unfortunately, the RHEL4 system for which  I was testing, doesn't like 
nfs-utils-1.0.6-65.
With nfs-utils-1.0.6-65, rpcgssd dies at
rpc.gssd[5626]: rpcsec_gss: in authgss_create_default()
RPC: AUTH_GSS upcall timed out.
Please check user daemon is running!

The 70 77 patchlevels both give permission denied, and the above rpcgssd 
messages.
With the newest patch, I had to symlink lib/libgssapi_krb5.so -> 
lib64/libgssapi_krb5.so

This a new, up2date RHEL4, all rpm versions seem to match that of the 
CentOS I tested.


-jim


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs