Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Martin Wege]

On Wed, May 14, 2025 at 1:55 PM NeilBrown <neil@brown.name> wrote:
>
> On Wed, 14 May 2025, Jeff Layton wrote:
> Ignoring source ports makes no sense at all unless you enforce some other
> authentication, like krb5 or TLS, or unless you *know* that there are no
> unprivileged processes running on any client machines.

I don't like to ruin that party, but this is NOT realistic.

1. Kerberos5 support is HARD to set up, and fragile because not all
distributions test it on a regular basis. Config is hard, not all
distributions support all kinds of encryption methods, and Redhat's
crusade&maintainer mobbing to promote sssd at the expense of other
solutions left users with a half broken, overcomplicated Windows
Active Directory clone called sssd, which is an insane overkill for
most scenarios.
gssproxy is also a constant source of pain - just Google for the
Debian bug reports.

Short: Lack of test coverage in distros, not working from time to
time, sssd and gssproxy are more of a problem than a solution.

It really only makes sense for very big sites and a support contract
which covers support and bug fixes for Kerberos5 in NFS+gssproxy.


2. TLS: Wanna make NFS even slower? Then use NFS with TLS.

NFS filesystem over TLS support then feels like working with molasses.

Exacerbated by Linux's crazy desire to only support hyper-secure
post-quantum encryption method (so no fast arcfour, because that is
"insecure", and everyone is expected to only work with AMD
Threadripper 3995WX), lack of good threading through the TLS eye of
the needle, and LACK of support in NFS clients.

Interoperability is also a big problem (nay, it's ZERO
interoperability), as this is basically a Linux kernel client/kernel server only
solution.
libtirpc doesn't support TLS, Ganesha doesn't support TLS, so yeah,
this is an issue, and not a solution.

Fazit: Supporting your argumentation with Kerberos5 or TLS is not gonna fly.

Thanks,
Martin

Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Rick Macklem]

On Wed, May 14, 2025 at 2:51 PM Martin Wege <martin.l.wege@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca.
>
>
> On Wed, May 14, 2025 at 1:55 PM NeilBrown <neil@brown.name> wrote:
> >
> > On Wed, 14 May 2025, Jeff Layton wrote:
> > Ignoring source ports makes no sense at all unless you enforce some other
> > authentication, like krb5 or TLS, or unless you *know* that there are no
> > unprivileged processes running on any client machines.
>
> I don't like to ruin that party, but this is NOT realistic.
>
> 1. Kerberos5 support is HARD to set up, and fragile because not all
> distributions test it on a regular basis. Config is hard, not all
> distributions support all kinds of encryption methods, and Redhat's
> crusade&maintainer mobbing to promote sssd at the expense of other
> solutions left users with a half broken, overcomplicated Windows
> Active Directory clone called sssd, which is an insane overkill for
> most scenarios.
> gssproxy is also a constant source of pain - just Google for the
> Debian bug reports.
>
> Short: Lack of test coverage in distros, not working from time to
> time, sssd and gssproxy are more of a problem than a solution.
>
> It really only makes sense for very big sites and a support contract
> which covers support and bug fixes for Kerberos5 in NFS+gssproxy.
I will note that it is possible to configure a client to use SP4_NONE and
do Kerberos mounts without needing a Kerberos ticket. This avoids the
hassle of creating keytab entries for clients.

Once you do that, all you really need is a KDC (either MIT or Heimdal
can be set up easily) and the creation of a service principal for the
server and putting it in a keytab on the server.
(This really is not hard to do. I won't comment on what Linux distros
provide, since I have no expertise.)

>
>
> 2. TLS: Wanna make NFS even slower? Then use NFS with TLS.
>
> NFS filesystem over TLS support then feels like working with molasses.
This is not my limited experience. I have two old Dell laptops and they
easily do NFS over TLS at wire speed for their 1Gbps net ports.
(They do use about 30% of a CPU for each active mount's TCP connection.)
For faster (10Gbps-->) net connections, I wouldn't be surprised that you will
see a performance hit without offload hardware, but I'm not sure I would
call it "molasses".

There is offload hardware out there, but I have no access to such.

I will agree that NFS over TLS does not make sense for a well controlled
LAN, but the university example may not be a well controlled LAN,
except maybe within a server machine room.

>
> Exacerbated by Linux's crazy desire to only support hyper-secure
> post-quantum encryption method (so no fast arcfour, because that is
> "insecure", and everyone is expected to only work with AMD
> Threadripper 3995WX), lack of good threading through the TLS eye of
> the needle, and LACK of support in NFS clients.
>
> Interoperability is also a big problem (nay, it's ZERO
> interoperability), as this is basically a Linux kernel client/kernel server only
> solution.
Actually, I could have said it was a FreeBSD only solution at one point in
time. Now, both Linux and FreeBSD clients do it.
And, although I'd like to say otherwise, there are not a lot of other
current clients out there. (Yes, I know you are involved in the resurrected
CITI Windows client, which is good to see.)
I'd like to see the Windows client learn how to do NFS over TLS, since
a Windows laptop would be an ideal example of a client that could
use NFS over TLS. (With it, it can access a NFSv4 server from pretty
well anywhere. It is also possible to put a username in an X.509 cert.
on the client so that it performs all RPCs as that user and avoids any
need for Kerberos or messing with passwd/group entries. When this
is done it can stick any old uid/gid in the RPC header, because they
are ignored anyhow.)

Would I like to see Apple do this? Yes.
Do I expect MacOS to get this any time soon. Nope, since they have
never even upgraded their client to 4.1.

As for the VMware client, I doubt there is much use for NFS over TLS
in it.

What other clients are there out there? (Hummingbird's, now called
OpenText's NFSv4.0 client. I was surprised to see it was still possible
to buy it. And it probably can be put in the same category as the MacOS one.)

As for the merits of reserved port #s.. I'm staying out of that one. I made the
mistake of going down that rabbit hole some years ago. (I'll admit there are
certain situations where a reserved port# restriction is useful. A system where
root is trusted, but the users are not, could be one.)

rick

> libtirpc doesn't support TLS, Ganesha doesn't support TLS, so yeah,
> this is an issue, and not a solution.
>
> Fazit: Supporting your argumentation with Kerberos5 or TLS is not gonna fly.
>
> Thanks,
> Martin
>

NFS/TLS situation on Windows - Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Lionel Cons]

On Thu, 15 May 2025 at 04:09, Rick Macklem <rick.macklem@gmail.com> wrote:
>
> On Wed, May 14, 2025 at 2:51 PM Martin Wege <martin.l.wege@gmail.com> wrote:
> What other clients are there out there? (Hummingbird's, now called
> OpenText's NFSv4.0 client. I was surprised to see it was still possible
> to buy it. And it probably can be put in the same category as the MacOS one.)

Situation on Windows:
1. OpenText NFSv4 client: We've talked to Opentext about TLS support,
and they do not know whether a market for NFS over TLS outside what
they call the "Linux bubble" will ever martialise. There is also risk,
both engineering and drastic performance degradation if the Windows
native TLS is used (this is a kernel driver, so only the Windows
native TLS can be used).
So this is not going to happen unless someone pays, and the
performance will not be great.

2. ms-nfs41-client NFSv4.2 client: I've talked to Roland Mainz, who is
working with Tigran Mkrtchyan on ms-nfs41-client (Windows NFSv4.2
client). Their RPC is based on libtirpc, and if steved-libtirpc gets
TLS support, then this can be easily ported. But Roland (didn't ask
Tigran yet) doesn't have time to implement TLS support in libtirpc.

3. Windows Server 20xx NFSv4.1 server: Other department went through a
cycle of meetings with Microsoft in 2024/2025, so far Microsoft wants
"market demand" (which doesn't seem to materialise), and cautions that
the performance might be below 50% of a similar SMB configuration,
because TLS is not considered to be a good match for network
filesystems.

Summary:
Forget about NFS/TLS on Windows.

Lionel

Re: NFS/TLS situation on Windows - Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Rick Macklem]

On Mon, May 19, 2025 at 6:03 AM Lionel Cons <lionelcons1972@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca.
>
>
> On Thu, 15 May 2025 at 04:09, Rick Macklem <rick.macklem@gmail.com> wrote:
> >
> > On Wed, May 14, 2025 at 2:51 PM Martin Wege <martin.l.wege@gmail.com> wrote:
> > What other clients are there out there? (Hummingbird's, now called
> > OpenText's NFSv4.0 client. I was surprised to see it was still possible
> > to buy it. And it probably can be put in the same category as the MacOS one.)
>
> Situation on Windows:
> 1. OpenText NFSv4 client: We've talked to Opentext about TLS support,
> and they do not know whether a market for NFS over TLS outside what
> they call the "Linux bubble" will ever martialise. There is also risk,
> both engineering and drastic performance degradation if the Windows
> native TLS is used (this is a kernel driver, so only the Windows
> native TLS can be used).
> So this is not going to happen unless someone pays, and the
> performance will not be great.
>
> 2. ms-nfs41-client NFSv4.2 client: I've talked to Roland Mainz, who is
> working with Tigran Mkrtchyan on ms-nfs41-client (Windows NFSv4.2
> client). Their RPC is based on libtirpc, and if steved-libtirpc gets
> TLS support, then this can be easily ported. But Roland (didn't ask
> Tigran yet) doesn't have time to implement TLS support in libtirpc.
>
> 3. Windows Server 20xx NFSv4.1 server: Other department went through a
> cycle of meetings with Microsoft in 2024/2025, so far Microsoft wants
> "market demand" (which doesn't seem to materialise), and cautions that
> the performance might be below 50% of a similar SMB configuration,
> because TLS is not considered to be a good match for network
> filesystems.
>
> Summary:
> Forget about NFS/TLS on Windows.
Well, for #1 and #3 I'm not surprised and would agree.
I would like to find a way forward for #2.
I will take a look at the libtirpc sources and try and cobble to-gether
a prototype using the OpenSSL libraries.

I am not sure how helpful that will be for Roland, but it might be a
starting point. (It depends upon what Microsoft provides in the kernel
w.r.t. TLS and whether the driver uses a libtirpc library built from sources.

I do plan on posting to the mailing list for #2, since I did a short
test drive of the driver.

rick

>
> Lionel
>

Re: NFS/TLS situation on Windows - Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Lionel Cons]

On Tue, 20 May 2025 at 03:38, Rick Macklem <rick.macklem@gmail.com> wrote:
>
> On Mon, May 19, 2025 at 6:03 AM Lionel Cons <lionelcons1972@gmail.com> wrote:
> >
> > CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca.
> >
> >
> > On Thu, 15 May 2025 at 04:09, Rick Macklem <rick.macklem@gmail.com> wrote:
> > >
> > > On Wed, May 14, 2025 at 2:51 PM Martin Wege <martin.l.wege@gmail.com> wrote:
> > > What other clients are there out there? (Hummingbird's, now called
> > > OpenText's NFSv4.0 client. I was surprised to see it was still possible
> > > to buy it. And it probably can be put in the same category as the MacOS one.)
> >
> > Situation on Windows:
> > 1. OpenText NFSv4 client: We've talked to Opentext about TLS support,
> > and they do not know whether a market for NFS over TLS outside what
> > they call the "Linux bubble" will ever martialise. There is also risk,
> > both engineering and drastic performance degradation if the Windows
> > native TLS is used (this is a kernel driver, so only the Windows
> > native TLS can be used).
> > So this is not going to happen unless someone pays, and the
> > performance will not be great.
> >
> > 2. ms-nfs41-client NFSv4.2 client: I've talked to Roland Mainz, who is
> > working with Tigran Mkrtchyan on ms-nfs41-client (Windows NFSv4.2
> > client). Their RPC is based on libtirpc, and if steved-libtirpc gets
> > TLS support, then this can be easily ported. But Roland (didn't ask
> > Tigran yet) doesn't have time to implement TLS support in libtirpc.
> >
> > 3. Windows Server 20xx NFSv4.1 server: Other department went through a
> > cycle of meetings with Microsoft in 2024/2025, so far Microsoft wants
> > "market demand" (which doesn't seem to materialise), and cautions that
> > the performance might be below 50% of a similar SMB configuration,
> > because TLS is not considered to be a good match for network
> > filesystems.
> >
> > Summary:
> > Forget about NFS/TLS on Windows.
> Well, for #1 and #3 I'm not surprised and would agree.
> I would like to find a way forward for #2.
> I will take a look at the libtirpc sources and try and cobble to-gether
> a prototype using the OpenSSL libraries.

That would be awesome

>
> I am not sure how helpful that will be for Roland, but it might be a
> starting point. (It depends upon what Microsoft provides in the kernel
> w.r.t. TLS and whether the driver uses a libtirpc library built from sources.

If it works with steved-libtirpc on Linux, then it should work with
the libtirpc from ms-nfs41-client too.

Question is whether the openssl license is compatible to LGPL used by
ms-nfs41-client

>
> I do plan on posting to the mailing list for #2, since I did a short
> test drive of the driver.

Thank you



Lionel

Re: NFS/TLS situation on Windows - Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Rick Macklem]

On Wed, May 21, 2025 at 12:07 AM Lionel Cons <lionelcons1972@gmail.com> wrote:
>
> On Tue, 20 May 2025 at 03:38, Rick Macklem <rick.macklem@gmail.com> wrote:
> >
> > On Mon, May 19, 2025 at 6:03 AM Lionel Cons <lionelcons1972@gmail.com> wrote:
> > >
> > > CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca.
> > >
> > >
> > > On Thu, 15 May 2025 at 04:09, Rick Macklem <rick.macklem@gmail.com> wrote:
> > > >
> > > > On Wed, May 14, 2025 at 2:51 PM Martin Wege <martin.l.wege@gmail.com> wrote:
> > > > What other clients are there out there? (Hummingbird's, now called
> > > > OpenText's NFSv4.0 client. I was surprised to see it was still possible
> > > > to buy it. And it probably can be put in the same category as the MacOS one.)
> > >
> > > Situation on Windows:
> > > 1. OpenText NFSv4 client: We've talked to Opentext about TLS support,
> > > and they do not know whether a market for NFS over TLS outside what
> > > they call the "Linux bubble" will ever martialise. There is also risk,
> > > both engineering and drastic performance degradation if the Windows
> > > native TLS is used (this is a kernel driver, so only the Windows
> > > native TLS can be used).
> > > So this is not going to happen unless someone pays, and the
> > > performance will not be great.
> > >
> > > 2. ms-nfs41-client NFSv4.2 client: I've talked to Roland Mainz, who is
> > > working with Tigran Mkrtchyan on ms-nfs41-client (Windows NFSv4.2
> > > client). Their RPC is based on libtirpc, and if steved-libtirpc gets
> > > TLS support, then this can be easily ported. But Roland (didn't ask
> > > Tigran yet) doesn't have time to implement TLS support in libtirpc.
> > >
> > > 3. Windows Server 20xx NFSv4.1 server: Other department went through a
> > > cycle of meetings with Microsoft in 2024/2025, so far Microsoft wants
> > > "market demand" (which doesn't seem to materialise), and cautions that
> > > the performance might be below 50% of a similar SMB configuration,
> > > because TLS is not considered to be a good match for network
> > > filesystems.
> > >
> > > Summary:
> > > Forget about NFS/TLS on Windows.
> > Well, for #1 and #3 I'm not surprised and would agree.
> > I would like to find a way forward for #2.
> > I will take a look at the libtirpc sources and try and cobble to-gether
> > a prototype using the OpenSSL libraries.
>
> That would be awesome
>
> >
> > I am not sure how helpful that will be for Roland, but it might be a
> > starting point. (It depends upon what Microsoft provides in the kernel
> > w.r.t. TLS and whether the driver uses a libtirpc library built from sources.
>
> If it works with steved-libtirpc on Linux, then it should work with
> the libtirpc from ms-nfs41-client too.
Hmm. Maybe I will put the function calls that would do upcalls in and
then write a simple set of them that do the OpenSSL calls. I need to
find out more about how the driver does RPCs.

I'll try to look at the sources on github.

>
> Question is whether the openssl license is compatible to LGPL used by
> ms-nfs41-client
I don't see why LGPL'd code cannot link to non-GPL'd open source, but
I'm no lawyer. (When this is done, it will only be attractive to those that
want RPC-over-TLS, since the TLS libraries will need to be linked into
the app. using the libtirpc, I think? (I don't understand library versioning,
so I don't know if there is a way to avoid this when apps. don't want
RPC-over-TLS?)

The question is what TLS libraries are normally used for Windows?

rick

>
> >
> > I do plan on posting to the mailing list for #2, since I did a short
> > test drive of the driver.
>
> Thank you
>
>
>
> Lionel

Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Chuck Lever]

On 5/14/25 5:50 PM, Martin Wege wrote:
> On Wed, May 14, 2025 at 1:55 PM NeilBrown <neil@brown.name> wrote:
>>
>> On Wed, 14 May 2025, Jeff Layton wrote:
>> Ignoring source ports makes no sense at all unless you enforce some other
>> authentication, like krb5 or TLS, or unless you *know* that there are no
>> unprivileged processes running on any client machines.
> 
> I don't like to ruin that party, but this is NOT realistic.
> 
> 1. Kerberos5 support is HARD to set up, and fragile because not all
> distributions test it on a regular basis. Config is hard, not all
> distributions support all kinds of encryption methods, and Redhat's
> crusade&maintainer mobbing to promote sssd at the expense of other
> solutions left users with a half broken, overcomplicated Windows
> Active Directory clone called sssd, which is an insane overkill for
> most scenarios.
> gssproxy is also a constant source of pain - just Google for the
> Debian bug reports.
> 
> Short: Lack of test coverage in distros, not working from time to
> time, sssd and gssproxy are more of a problem than a solution.
> 
> It really only makes sense for very big sites and a support contract
> which covers support and bug fixes for Kerberos5 in NFS+gssproxy.

Brief general comment: We are well aware of the administrative
challenges presented by Kerberos. :-)


> 2. TLS: Wanna make NFS even slower? Then use NFS with TLS.
> 
> NFS filesystem over TLS support then feels like working with molasses.

We'd like to hear quantitative evidence. In general, TLS with a NIC
that has encryption offload is going to be faster than NFS/Kerberos with
the privacy service. krb5p cannot be offloaded, full stop.

An increasing number of encryption-capable NICs are reaching the
marketplace, and the selection of encryption algorithms available for
TLS includes some CPU-efficient choices.

Thus our expectation is that TLS will become a more performant
solution than Kerberos. In addition, the trend is towards always-on
encryption (QUICv1). IMO you will not be able to avoid encryption-in-
transit in the future.


> Exacerbated by Linux's crazy desire to only support hyper-secure
> post-quantum encryption method (so no fast arcfour, because that is
> "insecure", and everyone is expected to only work with AMD
> Threadripper 3995WX), lack of good threading through the TLS eye of
> the needle, and LACK of support in NFS clients.

I believe the IETF has also broadly discouraged the use of easy-to-
defeat encryption algorithms. Perhaps this desire is not limited to only
Linux.

Using a deprecated encryption algorithm means you get very little
real security in addition to worse performance, so it's not a good
choice.


> Interoperability is also a big problem (nay, it's ZERO
> interoperability), as this is basically a Linux kernel client/kernel server only
> solution.
> libtirpc doesn't support TLS, Ganesha doesn't support TLS, so yeah,
> this is an issue, and not a solution.
> 
> Fazit: Supporting your argumentation with Kerberos5 or TLS is not gonna fly.

I don't think Jeff was suggesting that everyone can just switch
to using cryptography-based security. The point is that real security is
not provided by a cleartext 32-bit word in a network header, and we
should not continue pretending that it is.


-- 
Chuck Lever

Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Martin Wege]

On Thu, May 15, 2025 at 2:29 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>
> On 5/14/25 5:50 PM, Martin Wege wrote:
> > On Wed, May 14, 2025 at 1:55 PM NeilBrown <neil@brown.name> wrote:
> >>
> >> On Wed, 14 May 2025, Jeff Layton wrote:
> >> Ignoring source ports makes no sense at all unless you enforce some other
> >> authentication, like krb5 or TLS, or unless you *know* that there are no
> >> unprivileged processes running on any client machines.
> >
> > I don't like to ruin that party, but this is NOT realistic.
> >
> > 1. Kerberos5 support is HARD to set up, and fragile because not all
> > distributions test it on a regular basis. Config is hard, not all
> > distributions support all kinds of encryption methods, and Redhat's
> > crusade&maintainer mobbing to promote sssd at the expense of other
> > solutions left users with a half broken, overcomplicated Windows
> > Active Directory clone called sssd, which is an insane overkill for
> > most scenarios.
> > gssproxy is also a constant source of pain - just Google for the
> > Debian bug reports.
> >
> > Short: Lack of test coverage in distros, not working from time to
> > time, sssd and gssproxy are more of a problem than a solution.
> >
> > It really only makes sense for very big sites and a support contract
> > which covers support and bug fixes for Kerberos5 in NFS+gssproxy.
>
> Brief general comment: We are well aware of the administrative
> challenges presented by Kerberos. :-)

Well, the primary roadblock for development and TESTING seems to be
the absolute insane setup requirements. Every doc I find talks about
sssd, LDAP, Krb5.

I think what is needed for small scale testing is the so called
"grocery shop setup":
Two machines with local accounts (i.e. /etc/passwd), one NFS server
with Krb5 server, one NFS client with Krb5 client.
NO LDAP, NO TLS, NO Krb5 preauth

Unfortunately all docs describing that are GONE.

>
>
> > 2. TLS: Wanna make NFS even slower? Then use NFS with TLS.
> >
> > NFS filesystem over TLS support then feels like working with molasses.
>
> We'd like to hear quantitative evidence. In general, TLS with a NIC
> that has encryption offload is going to be faster than NFS/Kerberos with
> the privacy service.

You can test it yourself:
1. WAN config with 0.1s latency per hop, 2 hops, leads to "simple"
tasks like mkdir, mkfile, mknod to take > 1.6 seconds.
2. Parallelism is ruined by all traffic going through the TLS layer.
3. Latency is unbearable high, and [2] is not going to save the
situation this time (never was a solution anyway)
3. TLS layer does not respect RPC boundaries, a problem shared with
ssh and other encryption and compression programs. Simply said, TLS
has no concept or api similar to TCP CORK, or just flush. RPC packages
get "stuck" in the TLS layer, leading to excessive waiting times until
more data arrives. This might even not be fixable without a better TLS
protocol, or at least an API which handles message boundaries.

>  krb5p cannot be offloaded, full stop.

Sigh.

Who is claiming that? Same marketing gurus who "foresaw the rise of the TLS"?

>
> An increasing number of encryption-capable NICs are reaching the
> marketplace, and the selection of encryption algorithms available for
> TLS includes some CPU-efficient choices.

Poinnt [3]. TLS is just very poorly suited for RPC, and no hardware
acceleration will fix that.

TLS is designed for large chunk transfers. All the tiny bitsy RPC
packages don't work well over TLS, unless there is a lot of parallel
traffic, or large READ or WRITE transfers.

>
> Thus our expectation is that TLS will become a more performant
> solution than Kerberos.

Which marketing team came up with that "prediction"?

Judging from WAN experience, judging from the design and API deficits,
NFS over TLS is simply a failed experiment.

Just some statistics:
Our company is consulting 86 companies trying to deploy NFS over TLS
since the beginning of 2024, and NONE of them thinks it is usable,
ready for production or even close to something like that. Everyone is
pretty upset about latency spikes, lack of throughput, very high CPU
usage and other problems.

> In addition, the trend is towards always-on
> encryption (QUICv1). IMO you will not be able to avoid encryption-in-
> transit in the future.
>
>
> > Exacerbated by Linux's crazy desire to only support hyper-secure
> > post-quantum encryption method (so no fast arcfour, because that is
> > "insecure", and everyone is expected to only work with AMD
> > Threadripper 3995WX), lack of good threading through the TLS eye of
> > the needle, and LACK of support in NFS clients.
>
> I believe the IETF has also broadly discouraged the use of easy-to-
> defeat encryption algorithms.

Sometimes I think the IETF decisions are a) from Mars or b) done by
marketing, and not rooted in reality.

> Perhaps this desire is not limited to only
> Linux.

And how should the "small shops" (below what Oracle would call
multi-million enterprises) should deploy an usable NFS over TLS
configuration then? Basically you force them to either have ZERO
security, or a security they CANNOT AFFORD.
No middle ground.

Which brings us back to the debate of the secure/insecure export
option, and overkill Krb5 configs like sssd.

>
> Using a deprecated encryption algorithm means you get very little
> real security in addition to worse performance, so it's not a good
> choice.

Then please forget about using TLS. and don't recommend a
broken-by-design solution.

Thanks,
Martin

Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Rick Macklem]

On Sun, May 18, 2025 at 1:00 PM Martin Wege <martin.l.wege@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca.
>
>
> On Thu, May 15, 2025 at 2:29 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> >
> > On 5/14/25 5:50 PM, Martin Wege wrote:
> > > On Wed, May 14, 2025 at 1:55 PM NeilBrown <neil@brown.name> wrote:
> > >>
> > >> On Wed, 14 May 2025, Jeff Layton wrote:
> > >> Ignoring source ports makes no sense at all unless you enforce some other
> > >> authentication, like krb5 or TLS, or unless you *know* that there are no
> > >> unprivileged processes running on any client machines.
> > >
> > > I don't like to ruin that party, but this is NOT realistic.
> > >
> > > 1. Kerberos5 support is HARD to set up, and fragile because not all
> > > distributions test it on a regular basis. Config is hard, not all
> > > distributions support all kinds of encryption methods, and Redhat's
> > > crusade&maintainer mobbing to promote sssd at the expense of other
> > > solutions left users with a half broken, overcomplicated Windows
> > > Active Directory clone called sssd, which is an insane overkill for
> > > most scenarios.
> > > gssproxy is also a constant source of pain - just Google for the
> > > Debian bug reports.
> > >
> > > Short: Lack of test coverage in distros, not working from time to
> > > time, sssd and gssproxy are more of a problem than a solution.
> > >
> > > It really only makes sense for very big sites and a support contract
> > > which covers support and bug fixes for Kerberos5 in NFS+gssproxy.
> >
> > Brief general comment: We are well aware of the administrative
> > challenges presented by Kerberos. :-)
>
> Well, the primary roadblock for development and TESTING seems to be
> the absolute insane setup requirements. Every doc I find talks about
> sssd, LDAP, Krb5.
>
> I think what is needed for small scale testing is the so called
> "grocery shop setup":
> Two machines with local accounts (i.e. /etc/passwd), one NFS server
> with Krb5 server, one NFS client with Krb5 client.
> NO LDAP, NO TLS, NO Krb5 preauth
>
> Unfortunately all docs describing that are GONE.
>
> >
> >
> > > 2. TLS: Wanna make NFS even slower? Then use NFS with TLS.
> > >
> > > NFS filesystem over TLS support then feels like working with molasses.
> >
> > We'd like to hear quantitative evidence. In general, TLS with a NIC
> > that has encryption offload is going to be faster than NFS/Kerberos with
> > the privacy service.
>
> You can test it yourself:
> 1. WAN config with 0.1s latency per hop, 2 hops, leads to "simple"
> tasks like mkdir, mkfile, mknod to take > 1.6 seconds.
It doesn't matter if you use TLS or not, a 100msec transit delay is going
to result in abysmal performance.
Btw, last week I was doing remote testing during the Bakeathon from
the centre of the universe (Valemount, BC. A village with a population
about 1000
3 time zones away from Ann Arbor). I got a ping RTT of about 60-70msec,
so a lot better than what you listed above. But, yes, performance was not
terrific. However, with or without TLS doesn't matter for this case.

Try testing with/without TLS when your client->server ping RTT is a
reasonable value.

> 2. Parallelism is ruined by all traffic going through the TLS layer.
I don't see why TLS will make any difference w.r.t. this. Each RPC usually
(not always) ends up as a separate TLS data record. Any serialization is
just the same as a TCP connection without TLS, from what I can see.

> 3. Latency is unbearable high, and [2] is not going to save the
> situation this time (never was a solution anyway)
Yes, but why would you assume this is used in such an environment?
(Try Cifs/SMB over connection with a 100msec transit delay (200msec RTT)
and see how well it performs.)
Maybe someone can try pinging a server "across town" and see what the
RTT is? Also, no one said this was for WAN environments.

Performance (or lack thereof) across WANs is another topic.

> 3. TLS layer does not respect RPC boundaries, a problem shared with
> ssh and other encryption and compression programs.
I cannot comment w.r.t. other implementations, but for FreeBSD, each RPC
message is typically a separate TLS data record, as I noted above.
(Once in a while, two RPC messages may get queued on TCP send queue
such that the KTLS sends them as one TLS data record, but it does not
delay waiting for more data than 1 RPC message before sending a TLS
data record.)

> Simply said, TLS
> has no concept or api similar to TCP CORK, or just flush. RPC packages
> get "stuck" in the TLS layer, leading to excessive waiting times until
> more data arrives. This might even not be fixable without a better TLS
> protocol, or at least an API which handles message boundaries.
It's not a problem for the TLS protocol (a data record can be any size
up to 14K, if I recall correctly. It might be an issue for some TLS
implementation
that chooses to wait for "more data" before sending a small TLS data record.

>
> >  krb5p cannot be offloaded, full stop.
>
> Sigh.
>
> Who is claiming that? Same marketing gurus who "foresaw the rise of the TLS"?
Well, RPCSEC_GSS using privacy (which is what krb5p is) only encrypts the data
portion of the RPC message and not the header.
To offload it, the offload hardware would need to know the exact byte
ranges within
the RPC messages to encrypt/decrypt. It isn't going to happen unless the whole
world adopts RPCSEC_GSS with privacy using Kerberos mechanism.

TLS on the other hand gets used quite a but. When was the last time you accessed
a web site with "http" and not "https"?

>
> >
> > An increasing number of encryption-capable NICs are reaching the
> > marketplace, and the selection of encryption algorithms available for
> > TLS includes some CPU-efficient choices.
>
> Poinnt [3]. TLS is just very poorly suited for RPC, and no hardware
> acceleration will fix that.
Maybe. I have not been able to get any experience with offload hardware.
(I have heard that the QAT stuff does require significant setup time, so I
suspect that it does not help w.r.t. NFS over TLS, which uses lots of
small TLS data records, but I have not seen any test results to confirm this.)

>
> TLS is designed for large chunk transfers. All the tiny bitsy RPC
> packages don't work well over TLS, unless there is a lot of parallel
> traffic, or large READ or WRITE transfers.
>
> >
> > Thus our expectation is that TLS will become a more performant
> > solution than Kerberos.
>
> Which marketing team came up with that "prediction"?
>
> Judging from WAN experience, judging from the design and API deficits,
> NFS over TLS is simply a failed experiment.
>
> Just some statistics:
> Our company is consulting 86 companies trying to deploy NFS over TLS
> since the beginning of 2024, and NONE of them thinks it is usable,
> ready for production or even close to something like that. Everyone is
> pretty upset about latency spikes, lack of throughput, very high CPU
> usage and other problems.
I will note that without offload hardware, I see about 30% of one CPU core
being used when transferring a large file over a 1Gbps LAN connection
x86-64 with AESNI).
This is fine for clients, but means a server with a lot of NFS over TLS active
client connections will either need a lot of cores or hardware offload, I think?

>
> > In addition, the trend is towards always-on
> > encryption (QUICv1). IMO you will not be able to avoid encryption-in-
> > transit in the future.
> >
> >
> > > Exacerbated by Linux's crazy desire to only support hyper-secure
> > > post-quantum encryption method (so no fast arcfour, because that is
> > > "insecure", and everyone is expected to only work with AMD
> > > Threadripper 3995WX), lack of good threading through the TLS eye of
> > > the needle, and LACK of support in NFS clients.
> >
> > I believe the IETF has also broadly discouraged the use of easy-to-
> > defeat encryption algorithms.
>
> Sometimes I think the IETF decisions are a) from Mars or b) done by
> marketing, and not rooted in reality.
I know Elon Musk wants to go to Mars, but I don't think he participates in
IETF meetings.

>
> > Perhaps this desire is not limited to only
> > Linux.
>
> And how should the "small shops" (below what Oracle would call
> multi-million enterprises) should deploy an usable NFS over TLS
> configuration then? Basically you force them to either have ZERO
> security, or a security they CANNOT AFFORD.
I deployed it here at home with nothing except command line openssl.
(There are easier ways to do a site local CA, but it works.)

You can look at:
https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt
or for Kerberos
https://people.freebsd.org/~rmacklem/nfs-krb5-setup.txt

rick

> No middle ground.
>
> Which brings us back to the debate of the secure/insecure export
> option, and overkill Krb5 configs like sssd.
>
> >
> > Using a deprecated encryption algorithm means you get very little
> > real security in addition to worse performance, so it's not a good
> > choice.
>
> Then please forget about using TLS. and don't recommend a
> broken-by-design solution.
>
> Thanks,
> Martin
>

Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Mkrtchyan, Tigran]

[-- Attachment #1: Type: text/plain, Size: 2515 bytes --]



----- Original Message -----
> From: "Martin Wege" <martin.l.wege@gmail.com>
> To: "Linux NFS Mailing List" <linux-nfs@vger.kernel.org>
> Sent: Wednesday, 14 May, 2025 23:50:00
> Subject: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the
> default for all exports

> On Wed, May 14, 2025 at 1:55 PM NeilBrown <neil@brown.name> wrote:
>>
>> On Wed, 14 May 2025, Jeff Layton wrote:
>> Ignoring source ports makes no sense at all unless you enforce some other
>> authentication, like krb5 or TLS, or unless you *know* that there are no
>> unprivileged processes running on any client machines.
> 
> I don't like to ruin that party, but this is NOT realistic.
> 
> 1. Kerberos5 support is HARD to set up, and fragile because not all
> distributions test it on a regular basis. Config is hard, not all
> distributions support all kinds of encryption methods, and Redhat's
> crusade&maintainer mobbing to promote sssd at the expense of other
> solutions left users with a half broken, overcomplicated Windows
> Active Directory clone called sssd, which is an insane overkill for
> most scenarios.
> gssproxy is also a constant source of pain - just Google for the
> Debian bug reports.
> 
> Short: Lack of test coverage in distros, not working from time to
> time, sssd and gssproxy are more of a problem than a solution.
> 
> It really only makes sense for very big sites and a support contract
> which covers support and bug fixes for Kerberos5 in NFS+gssproxy.
> 
> 
> 2. TLS: Wanna make NFS even slower? Then use NFS with TLS.
> 
> NFS filesystem over TLS support then feels like working with molasses.
> 
> Exacerbated by Linux's crazy desire to only support hyper-secure
> post-quantum encryption method (so no fast arcfour, because that is
> "insecure", and everyone is expected to only work with AMD
> Threadripper 3995WX), lack of good threading through the TLS eye of
> the needle, and LACK of support in NFS clients.
> 
> Interoperability is also a big problem (nay, it's ZERO
> interoperability), as this is basically a Linux kernel client/kernel server only
> solution.

dCache (a-ka nfs4j) server supports TLS, FreeBSD, ... 

So, it's not a Linux-only solution.

Tigran.

> libtirpc doesn't support TLS, Ganesha doesn't support TLS, so yeah,
> this is an issue, and not a solution.
> 
> Fazit: Supporting your argumentation with Kerberos5 or TLS is not gonna fly.
> 
> Thanks,
> Martin

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2826 bytes --]

Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Dan Shelton]

On Wed, 14 May 2025 at 23:51, Martin Wege <martin.l.wege@gmail.com> wrote:
> Interoperability is also a big problem (nay, it's ZERO
> interoperability), as this is basically a Linux kernel client/kernel server only
> solution.
> libtirpc doesn't support TLS, Ganesha doesn't support TLS, so yeah,
> this is an issue, and not a solution.
>
> Fazit: Supporting your argumentation with Kerberos5 or TLS is not gonna fly.

I tried to add TLS support to libtirpc, but I think it's simply not
possible. The APIs are just not compatible.
Ganesha folks also tried the same, and failed - their ntirpc would
require a major redesign to support TLS.

So, why do the NFS folks even bother with NFS over TLS, if it is this
kind of mismatch?

Dan
-- 
Dan Shelton - Cluster Specialist Win/Lin/Bsd

Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Rick Macklem]

On Sun, May 18, 2025 at 7:15 PM Dan Shelton <dan.f.shelton@gmail.com> wrote:
>
> On Wed, 14 May 2025 at 23:51, Martin Wege <martin.l.wege@gmail.com> wrote:
> > Interoperability is also a big problem (nay, it's ZERO
> > interoperability), as this is basically a Linux kernel client/kernel server only
> > solution.
> > libtirpc doesn't support TLS, Ganesha doesn't support TLS, so yeah,
> > this is an issue, and not a solution.
> >
> > Fazit: Supporting your argumentation with Kerberos5 or TLS is not gonna fly.
>
> I tried to add TLS support to libtirpc, but I think it's simply not
> possible. The APIs are just not compatible.
> Ganesha folks also tried the same, and failed - their ntirpc would
> require a major redesign to support TLS.
Well, I'll admit I don't know if the userspace RPC code in FreeBSD
is considered the tirpc or not. (It's in libc and happens to have Frank's
initials on it from long ago when he worked on it for NetBSD.)

Anyhow, at a quick glance, it doesn't look all that difficult.

The low level functions in clnt_vc.c that do I/O on the socket
are called __msgread() and __msgwrite(). If RPC over TLS is
enabled, those functions would need to use SSL_read() and SSL_write().

When a new TCP connection is created (in clnt_vc_create() in this code),
it then needs to do a Null RPC with AUTH_TLS (7) as the authentication
flavor. If it gets STARTTLS in the reply's verifier, then it does a
SSL_connect(). If that succeeds, it flips some flag to tell the I/O
functions to use SSL_read() and SSL_write().
(There is some arm waving to create the SSL context and associate
the TCP socket with it. You can find all that in the sources for
rpc.tlsclntd.c in FreeBSD.)

I haven't actually done this (I did the daemons and krpc mods for
FreeBSD, but not libc ones), but unless there is some corner
that makes it really difficult, it doesn't look like a lot of work to me.
(If Frank or anyone else wants to do this, they can email me for
more info.)

rick
ps: The biggest hassle is that, once you do it, the applications that
      use the library will also need to link in a bunch of SSL
libraries as well.
      (The simple thing is to have a separate library version. Since it's in
       libc in FreeBSD, this is more bothersome.)

>
> So, why do the NFS folks even bother with NFS over TLS, if it is this
> kind of mismatch?
>
> Dan
> --
> Dan Shelton - Cluster Specialist Win/Lin/Bsd
>

RE: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Frank Filz]

> -----Original Message-----
> From: Dan Shelton [mailto:dan.f.shelton@gmail.com]
> Sent: Sunday, May 18, 2025 7:14 PM
> To: Linux NFS Mailing List <linux-nfs@vger.kernel.org>; libtirpc-
> devel@lists.sourceforge.net
> Subject: Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH
> nfs-utils] exportfs: make "insecure" the default for all exports
> 
> On Wed, 14 May 2025 at 23:51, Martin Wege <martin.l.wege@gmail.com>
> wrote:
> > Interoperability is also a big problem (nay, it's ZERO
> > interoperability), as this is basically a Linux kernel client/kernel
> > server only solution.
> > libtirpc doesn't support TLS, Ganesha doesn't support TLS, so yeah,
> > this is an issue, and not a solution.
> >
> > Fazit: Supporting your argumentation with Kerberos5 or TLS is not gonna fly.
> 
> I tried to add TLS support to libtirpc, but I think it's simply not possible. The APIs
> are just not compatible.
> Ganesha folks also tried the same, and failed - their ntirpc would require a major
> redesign to support TLS.

For what it's worth, we (Ganesha) are still working on offering TLS. There are options (including kTLS) that seem like they would work (mostly) for us, the issue is if there needs to be a rekeying negotiation.

I can see how once a TLS session is started, we could integrate a ktls socket into our ntirpc epoll loop and send and receive packets on the TLS session. But when rekeying is necessary, I don't quite see how to get back into a library to do that, particularly in a way that still uses our epoll loop. That is the same for user space libraries. We have looked at OpenSSL and s2n.

We do see a demand for protected RPC sessions for NFS and TLS seems to be the best option at the moment.

Frank

Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH nfs-utils] exportfs: make "insecure" the default for all exports

[Chuck Lever]

On 5/19/25 2:44 PM, Frank Filz wrote:
> 
> 
>> -----Original Message-----
>> From: Dan Shelton [mailto:dan.f.shelton@gmail.com]
>> Sent: Sunday, May 18, 2025 7:14 PM
>> To: Linux NFS Mailing List <linux-nfs@vger.kernel.org>; libtirpc-
>> devel@lists.sourceforge.net
>> Subject: Re: Why TLS and Kerberos are not useful for NFS security Re: [PATCH
>> nfs-utils] exportfs: make "insecure" the default for all exports
>>
>> On Wed, 14 May 2025 at 23:51, Martin Wege <martin.l.wege@gmail.com>
>> wrote:
>>> Interoperability is also a big problem (nay, it's ZERO
>>> interoperability), as this is basically a Linux kernel client/kernel
>>> server only solution.
>>> libtirpc doesn't support TLS, Ganesha doesn't support TLS, so yeah,
>>> this is an issue, and not a solution.
>>>
>>> Fazit: Supporting your argumentation with Kerberos5 or TLS is not gonna fly.
>>
>> I tried to add TLS support to libtirpc, but I think it's simply not possible. The APIs
>> are just not compatible.
>> Ganesha folks also tried the same, and failed - their ntirpc would require a major
>> redesign to support TLS.
> 
> For what it's worth, we (Ganesha) are still working on offering TLS. There are options (including kTLS) that seem like they would work (mostly) for us, the issue is if there needs to be a rekeying negotiation.
> 
> I can see how once a TLS session is started, we could integrate a ktls socket into our ntirpc epoll loop and send and receive packets on the TLS session. But when rekeying is necessary, I don't quite see how to get back into a library to do that, particularly in a way that still uses our epoll loop. That is the same for user space libraries. We have looked at OpenSSL and s2n.

The current behavior for our implementation is that when re-keying is
needed, the connection is closed and a fresh handshake is done. This
does not appear to happen often.

KeyUpdate support was added to ktls recently, but we haven't plumbed
it into the Linux kernel RPC implementation.


> We do see a demand for protected RPC sessions for NFS and TLS seems to be the best option at the moment.



-- 
Chuck Lever