[NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Frank Steiner]

Hi,

I'm fighting with my firewall to get nfs-over-tcp through. 

The server is outside the firewall, the client is inside. The firewall
allows all tcp back-connections (without syn), no UDPs. Mount on the
client side worked fine with kernel 2.6.16 in SLES 10.

Now when the NFS client is running SLES 11 with its kernel 2.6.27, 
the NFS server tries to make UDP connections from its ports 111 and 
700 to different ports on the client.

If the client is running SLES 10 with 2.6.16, those connections are
not tried from the server, no matter if the server runs 2.6.16 or
2.6.27.

So I've two questions:
1) Should nfs-over-tcp still use any UDP ports at all?
2) What has been changed in the client code between 2.6.16 and 2.6.27
   that could cause this behaviour?

Is there a way to prevent those UDP connects?

cu,
Frank

-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Leonardo Chiquitto]

> Now when the NFS client is running SLES 11 with its kernel 2.6.27,
> the NFS server tries to make UDP connections from its ports 111 and
> 700 to different ports on the client.
>
> If the client is running SLES 10 with 2.6.16, those connections are
> not tried from the server, no matter if the server runs 2.6.16 or
> 2.6.27.
>
> So I've two questions:
> 1) Should nfs-over-tcp still use any UDP ports at all?
> 2) What has been changed in the client code between 2.6.16 and 2.6.27
>   that could cause this behaviour?

By default it is now using UDP to connect to mountd.

> Is there a way to prevent those UDP connects?

Yes, add "mountproto=tcp" to your mount options.

Regards,
Leonardo

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Frank Steiner]

Leonardo Chiquitto wrote

>> Is there a way to prevent those UDP connects?
> 
> Yes, add "mountproto=tcp" to your mount options.

I was only aware of "proto=tcp" and indeed missed the mountproto option.

Thanks both Leonardo and Trond for your answers!

cu,
Frank

-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Tom Talpey]

At 11:26 AM 5/7/2009, Frank Steiner wrote:
>Leonardo Chiquitto wrote
>
>>> Is there a way to prevent those UDP connects?
>> 
>> Yes, add "mountproto=tcp" to your mount options.
>
>I was only aware of "proto=tcp" and indeed missed the mountproto option.
>
>Thanks both Leonardo and Trond for your answers!
>

There is one small caveat to using mountproto=tcp through firewalls:
while the mount will succeed, there are some side protocol exchanges
which may not.

In particular, if you do NLM file locking, there is a server callback (NLM
"granted") which the server may choose to issue via UDP. If this callback
is not seen by the client due to firewall blocking, there may be a 30-second
pause before a client retry unblocks the caller.

Also, the NSM (status monitor) exchanges are often performed via UDP.
Again, if you are using NLM and the server reboots, the client may not
become aware of this promptly, and lock reclaim will be affected.

OTOH, if your applications don't use locking on the NFS mounts, you'll
probably be fine.

Tom.


------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Aaron Wiebe]

On Thu, May 7, 2009 at 11:35 AM, Tom Talpey <tmtalpey@gmail.com> wrote:
>
> There is one small caveat to using mountproto=tcp through firewalls:
> while the mount will succeed, there are some side protocol exchanges
> which may not.
>
> In particular, if you do NLM file locking, there is a server callback (NLM
> "granted") which the server may choose to issue via UDP. If this callback
> is not seen by the client due to firewall blocking, there may be a 30-second
> pause before a client retry unblocks the caller.
>
> Also, the NSM (status monitor) exchanges are often performed via UDP.
> Again, if you are using NLM and the server reboots, the client may not
> become aware of this promptly, and lock reclaim will be affected.

Sorry for the slight threadjack, but a question along those lines...

My understanding is that currently portmap will not bind any UDP NLM
listeners unless there are UDP mounts on the machine.  I know there
are servers out there that will always speak NLM over UDP
(netapp/ontap being the prominent one), and as a result there can be
problems without firewalls.  If servers are out there that will speak
NLM over UDP regardless of the mount itself, shouldn't we be binding
NLM/UDP regardless of the mount transport?

(Or did I miss this change being reverted a while back?)

-Aaron

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Chuck Lever]

On May 7, 2009, at 12:08 PM, Aaron Wiebe wrote:
> On Thu, May 7, 2009 at 11:35 AM, Tom Talpey <tmtalpey@gmail.com>  
> wrote:
>>
>> There is one small caveat to using mountproto=tcp through firewalls:
>> while the mount will succeed, there are some side protocol exchanges
>> which may not.
>>
>> In particular, if you do NLM file locking, there is a server  
>> callback (NLM
>> "granted") which the server may choose to issue via UDP. If this  
>> callback
>> is not seen by the client due to firewall blocking, there may be a  
>> 30-second
>> pause before a client retry unblocks the caller.
>>
>> Also, the NSM (status monitor) exchanges are often performed via UDP.
>> Again, if you are using NLM and the server reboots, the client may  
>> not
>> become aware of this promptly, and lock reclaim will be affected.
>
> Sorry for the slight threadjack, but a question along those lines...
>
> My understanding is that currently portmap will not bind any UDP NLM
> listeners unless there are UDP mounts on the machine.

It's not portmap... lockd decides which listeners to start.

>  I know there
> are servers out there that will always speak NLM over UDP
> (netapp/ontap being the prominent one), and as a result there can be
> problems without firewalls.  If servers are out there that will speak
> NLM over UDP regardless of the mount itself, shouldn't we be binding
> NLM/UDP regardless of the mount transport?
>
> (Or did I miss this change being reverted a while back?)

This change was reverted upstream; see commit 8c3916f4.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Tom Talpey]

At 12:42 PM 5/7/2009, Chuck Lever wrote:
>On May 7, 2009, at 12:08 PM, Aaron Wiebe wrote:
>> On Thu, May 7, 2009 at 11:35 AM, Tom Talpey <tmtalpey@gmail.com>  
>> wrote:
>>>
>>> There is one small caveat to using mountproto=tcp through firewalls:
>>> while the mount will succeed, there are some side protocol exchanges
>>> which may not.
>>>
>>> In particular, if you do NLM file locking, there is a server  
>>> callback (NLM
>>> "granted") which the server may choose to issue via UDP. If this  
>>> callback
>>> is not seen by the client due to firewall blocking, there may be a  
>>> 30-second
>>> pause before a client retry unblocks the caller.
>>>
>>> Also, the NSM (status monitor) exchanges are often performed via UDP.
>>> Again, if you are using NLM and the server reboots, the client may  
>>> not
>>> become aware of this promptly, and lock reclaim will be affected.
>>
>> Sorry for the slight threadjack, but a question along those lines...
>>
>> My understanding is that currently portmap will not bind any UDP NLM
>> listeners unless there are UDP mounts on the machine.
>
>It's not portmap... lockd decides which listeners to start.

Also, there is an important distinction between NLM and NLM *callbacks*.
The NLM client follows the NFS protocol selection in many client kernels,
i.e. if you mount with proto=tcp, you get both NFS and NLM over TCP.

The issue is NLM callbacks, which are used only in specific cases where
clients take blocking locks, which actually need to block due to being already
held by another client. The server replies over NLM (e.g. TCP) with an indication
that a callback will arive later. But when the other lock is released, the callback
comes on a second connection, initiated from the server back to the client and
not on the original NLM channel. To make matters worse, some servers only
ever perform the callback on UDP in order to simplify and reduce the overhead
required.

If this callback doesn't arrive at the client, or arrives in such a way that
it's not recognized (e.g traverses a NAT and therefore changes source IP),
then the client only wakes up on a timer. The long pauses can be a real
problem, and one which only arises occasionally - i.e. very hard to trace down.

Just something to be aware of... it's a day-one defect in the NLM protocol,
actually.

Tom.

>
>>  I know there
>> are servers out there that will always speak NLM over UDP
>> (netapp/ontap being the prominent one), and as a result there can be
>> problems without firewalls.  If servers are out there that will speak
>> NLM over UDP regardless of the mount itself, shouldn't we be binding
>> NLM/UDP regardless of the mount transport?
>>
>> (Or did I miss this change being reverted a while back?)
>
>This change was reverted upstream; see commit 8c3916f4.
>
>--
>Chuck Lever
>chuck[dot]lever[at]oracle[dot]com
>


------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Aaron Wiebe]

On Thu, May 7, 2009 at 1:08 PM, Tom Talpey <tmtalpey@gmail.com> wrote:
> At 12:42 PM 5/7/2009, Chuck Lever wrote:
>>It's not portmap... lockd decides which listeners to start.

Thanks Chuck and Tom - that clarifies it.

Take care,
-Aaron

> Also, there is an important distinction between NLM and NLM *callback=
s*.
> The NLM client follows the NFS protocol selection in many client kern=
els,
> i.e. if you mount with proto=3Dtcp, you get both NFS and NLM over TCP=
=2E
>
> The issue is NLM callbacks, which are used only in specific cases whe=
re
> clients take blocking locks, which actually need to block due to bein=
g already
> held by another client. The server replies over NLM (e.g. TCP) with a=
n indication
> that a callback will arive later. But when the other lock is released=
, the callback
> comes on a second connection, initiated from the server back to the c=
lient and
> not on the original NLM channel. To make matters worse, some servers =
only
> ever perform the callback on UDP in order to simplify and reduce the =
overhead
> required.
>
> If this callback doesn't arrive at the client, or arrives in such a w=
ay that
> it's not recognized (e.g traverses a NAT and therefore changes source=
 IP),
> then the client only wakes up on a timer. The long pauses can be a re=
al
> problem, and one which only arises occasionally - i.e. very hard to t=
race down.
>
> Just something to be aware of... it's a day-one defect in the NLM pro=
tocol,
> actually.
>
> Tom.
>
>>
>>> =A0I know there
>>> are servers out there that will always speak NLM over UDP
>>> (netapp/ontap being the prominent one), and as a result there can b=
e
>>> problems without firewalls. =A0If servers are out there that will s=
peak
>>> NLM over UDP regardless of the mount itself, shouldn't we be bindin=
g
>>> NLM/UDP regardless of the mount transport?
>>>
>>> (Or did I miss this change being reverted a while back?)
>>
>>This change was reverted upstream; see commit 8c3916f4.
>>
>>--
>>Chuck Lever
>>chuck[dot]lever[at]oracle[dot]com
>>
>
>

-----------------------------------------------------------------------=
-------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! You=
r
production scanning environment may not be a perfect world - but thanks=
 to
Kodak, there's a perfect scanner to get the job done! With the NEW KODA=
K i700
Series Scanner you'll get full speed at 300 dpi even with all image=20
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Frank Steiner]

Tom Talpey wrote


> In particular, if you do NLM file locking, there is a server callback (NLM
> "granted") which the server may choose to issue via UDP. If this callback
> is not seen by the client due to firewall blocking, there may be a 30-second
> pause before a client retry unblocks the caller.
> 
> Also, the NSM (status monitor) exchanges are often performed via UDP.
> Again, if you are using NLM and the server reboots, the client may not
> become aware of this promptly, and lock reclaim will be affected.
> 
> OTOH, if your applications don't use locking on the NFS mounts, you'll
> probably be fine.

We do use locking on nfs mounts, so I wonder what that would mean for the
firewall. Currently I see connections from the NFS server *from* port 700
and 111 (we've fixed mountd port to 700) to (it seems) arbitrary udp
ports on the NFS clients.

Would that be enough to allow those? Or could the source ports be arbitrary
with NLM, too? I.e., would we have to open all udp traffic from the NFS
servers to all the NFS clients?

cu,
Frank


-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Trond Myklebust]

On Fri, 2009-05-08 at 08:03 +0200, Frank Steiner wrote:
> Tom Talpey wrote
> 
> 
> > In particular, if you do NLM file locking, there is a server callback (NLM
> > "granted") which the server may choose to issue via UDP. If this callback
> > is not seen by the client due to firewall blocking, there may be a 30-second
> > pause before a client retry unblocks the caller.
> > 
> > Also, the NSM (status monitor) exchanges are often performed via UDP.
> > Again, if you are using NLM and the server reboots, the client may not
> > become aware of this promptly, and lock reclaim will be affected.
> > 
> > OTOH, if your applications don't use locking on the NFS mounts, you'll
> > probably be fine.
> 
> We do use locking on nfs mounts, so I wonder what that would mean for the
> firewall. Currently I see connections from the NFS server *from* port 700
> and 111 (we've fixed mountd port to 700) to (it seems) arbitrary udp
> ports on the NFS clients.
> 
> Would that be enough to allow those? Or could the source ports be arbitrary
> with NLM, too? I.e., would we have to open all udp traffic from the NFS
> servers to all the NFS clients?

Most NFS servers allow you to pin the ports used by the lockd service.
In Linux, the kernel boot parameters lockd.nlm_tcpport and
lockd.nlm_udpport will suffice to do it for you (see
linux/Documentation/kernel-parameters.txt).

Trond


------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Tom Talpey]

At 02:03 AM 5/8/2009, Frank Steiner wrote:
>Tom Talpey wrote
>
>
>> In particular, if you do NLM file locking, there is a server callback (NLM
>> "granted") which the server may choose to issue via UDP. If this callback
>> is not seen by the client due to firewall blocking, there may be a 30-second
>> pause before a client retry unblocks the caller.
>> 
>> Also, the NSM (status monitor) exchanges are often performed via UDP.
>> Again, if you are using NLM and the server reboots, the client may not
>> become aware of this promptly, and lock reclaim will be affected.
>> 
>> OTOH, if your applications don't use locking on the NFS mounts, you'll
>> probably be fine.
>
>We do use locking on nfs mounts, so I wonder what that would mean for the
>firewall. Currently I see connections from the NFS server *from* port 700
>and 111 (we've fixed mountd port to 700) to (it seems) arbitrary udp
>ports on the NFS clients.
>
>Would that be enough to allow those? Or could the source ports be arbitrary
>with NLM, too? I.e., would we have to open all udp traffic from the NFS
>servers to all the NFS clients?

The calls from the server to client port 111 are portmapper lookups, very likely
while setting up the NSM callback that the server performs when it boots up.
If the portmap call succeeds, then a second NSM call will occur, to whatever
port the client NSM is using (700 sounds reasonable, but it's not guaranteed).

NLM callbacks on blocking lock collision will be similar, but will happen any time
not just at server reboot. The server will resolve the client NLM daemon using
port 111, then perform a call to the result on whatever port is returned.

There are very likely many ways to make these calls work through a firewall,
but unfortunately not very many ways to guarantee it. The big problem is
that the callbacks will be nearly silent on failure, so success is hard to detect!

Depending on how criticial the application is, I would make different
recommendations. But the safest option from a reliability standpoint might be
to, yes, open up all UDP traffic. Ouch. There are other ways, but they're
likely to be more fragile, and not all server/client combinations might work
with a single solution.

You might want to think through the DNS issues of this firewall, too. Are
all the clients using hostnames in the same DNS domain as the server? And,
do any of them use DHCP or other dynamic assignment? You want to be
sure to use FQDNs as the client hostname(2), to ensure the server can
always resolve them properly.

I made a presentation on some of the NSM issues a few years back for
Connectathon, if you're brave :-) you can check out:
	http://www.connectathon.org/talks06/talpey-cthon06-nsm.pdf

The very best solution, by the way, would be to use NFSv4. It has no
side protocols, and therefore no UDP issue. It does have a callback
connection from the server to the client, but is done with TCP and is
configurable.

Tom.


------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Chuck Lever]

On May 11, 2009, at 10:59 AM, Tom Talpey wrote:
> At 02:03 AM 5/8/2009, Frank Steiner wrote:
>> Tom Talpey wrote
>>
>>> In particular, if you do NLM file locking, there is a server  
>>> callback (NLM
>>> "granted") which the server may choose to issue via UDP. If this  
>>> callback
>>> is not seen by the client due to firewall blocking, there may be a  
>>> 30-second
>>> pause before a client retry unblocks the caller.
>>>
>>> Also, the NSM (status monitor) exchanges are often performed via  
>>> UDP.
>>> Again, if you are using NLM and the server reboots, the client may  
>>> not
>>> become aware of this promptly, and lock reclaim will be affected.
>>>
>>> OTOH, if your applications don't use locking on the NFS mounts,  
>>> you'll
>>> probably be fine.
>>
>> We do use locking on nfs mounts, so I wonder what that would mean  
>> for the
>> firewall. Currently I see connections from the NFS server *from*  
>> port 700
>> and 111 (we've fixed mountd port to 700) to (it seems) arbitrary udp
>> ports on the NFS clients.
>>
>> Would that be enough to allow those? Or could the source ports be  
>> arbitrary
>> with NLM, too? I.e., would we have to open all udp traffic from the  
>> NFS
>> servers to all the NFS clients?
>
> The calls from the server to client port 111 are portmapper lookups,  
> very likely
> while setting up the NSM callback that the server performs when it  
> boots up.
> If the portmap call succeeds, then a second NSM call will occur, to  
> whatever
> port the client NSM is using (700 sounds reasonable, but it's not  
> guaranteed).
>
> NLM callbacks on blocking lock collision will be similar, but will  
> happen any time
> not just at server reboot. The server will resolve the client NLM  
> daemon using
> port 111, then perform a call to the result on whatever port is  
> returned.
>
> There are very likely many ways to make these calls work through a  
> firewall,
> but unfortunately not very many ways to guarantee it. The big  
> problem is
> that the callbacks will be nearly silent on failure, so success is  
> hard to detect!
>
> Depending on how criticial the application is, I would make different
> recommendations. But the safest option from a reliability standpoint  
> might be
> to, yes, open up all UDP traffic. Ouch. There are other ways, but  
> they're
> likely to be more fragile, and not all server/client combinations  
> might work
> with a single solution.

You can specify a fixed listener port for both lockd and statd on your  
client.  And of course rpcbind uses a well-known port number.  So you  
should be able to open just those ports in your firewall.

> You might want to think through the DNS issues of this firewall,  
> too. Are
> all the clients using hostnames in the same DNS domain as the  
> server? And,
> do any of them use DHCP or other dynamic assignment? You want to be
> sure to use FQDNs as the client hostname(2), to ensure the server can
> always resolve them properly.

It depends, though, on if the client and server are sending IP  
addresses or hostnames in their SM_NOTIFY requests.

The server will record the client's IP address in NLM requests and use  
that when sending NLM_GRANT requests.  The server will look up the  
client's caller_name string and try to use that to call the client  
back for SM_NOTIFY.  The mon_name the server uses in the SM_NOTIFY  
request will either be its IP address or a hostname.  Usually this  
will match the hostname that the client used to contact it on the  
mount command.

If the client is behind a firewall and uses a private DNS service, the  
server will probably not be able to resolve the client's caller_name.   
I suppose one way to work around this is to provide a reasonable IP  
address to hostname binding for the client in the server's /etc/ 
hosts.  Another way might be to ensure the server's kernel sends IP  
addresses and not hostnames to its statd.  See /proc/sys/fs/nfs/ 
nsm_use_hostnames.

> I made a presentation on some of the NSM issues a few years back for
> Connectathon, if you're brave :-) you can check out:
> 	http://www.connectathon.org/talks06/talpey-cthon06-nsm.pdf
>
> The very best solution, by the way, would be to use NFSv4. It has no
> side protocols, and therefore no UDP issue. It does have a callback
> connection from the server to the client, but is done with TCP and is
> configurable.
>
> Tom.
>
>
> ------------------------------------------------------------------------------
> The NEW KODAK i700 Series Scanners deliver under ANY circumstances!  
> Your
> production scanning environment may not be a perfect world - but  
> thanks to
> Kodak, there's a perfect scanner to get the job done! With the NEW  
> KODAK i700
> Series Scanner you'll get full speed at 300 dpi even with all image
> processing features enabled. http://p.sf.net/sfu/kodak-com
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
> _______________________________________________
> Please note that nfs@lists.sourceforge.net is being discontinued.
> Please subscribe to linux-nfs@vger.kernel.org instead.
>    http://vger.kernel.org/vger-lists.html#linux-nfs
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"  
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com




------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Frank Steiner]

Tom, thanks for all your explanations! I've got to go through them
carefully to make sure I understand every kind of side-effect with 
all the different protocols and callback-stuff etc.

NFSv4 could indeed be an option for us, so I will check that also,
and I'm also going to read your talk. 

Thanks a lot for caring!

cu,
Frank

-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Frank Steiner]

Tom Talpey wrote

> The very best solution, by the way, would be to use NFSv4. It has no
> side protocols, and therefore no UDP issue. It does have a callback
> connection from the server to the client, but is done with TCP and is
> configurable.

I've indeed switched our through-firewall-nfsservers to NFSv4 and
the problems are gone. Thanks a lot for pointing me there!
I only open port 2049/tcp and everything works.

However, I still see blocked connections on the firewall, coming from
the NFS client to the NFS server: 
...PROTO=TCP SPT=55598 DPT=111...
rpcinfo tells me the portmapper is running at port 111 (udp and tcp).

I didn't find a clear statement when googling if that should happen
with NFSv4 or not. It doesn't seem to block the NFS share in any way,
at least as far as I can see. 

I wouldn't mind to open tcp port 111 to the NFS server. I'm just curios
if that behaviour is correct or not with NFSv4.

cu,
Frank

-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment. 
http://p.sf.net/sfu/businessobjects
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Tom Talpey]

At 02:38 AM 5/15/2009, Frank Steiner wrote:
>Tom Talpey wrote
>
>> The very best solution, by the way, would be to use NFSv4. It has no
>> side protocols, and therefore no UDP issue. It does have a callback
>> connection from the server to the client, but is done with TCP and is
>> configurable.
>
>I've indeed switched our through-firewall-nfsservers to NFSv4 and
>the problems are gone. Thanks a lot for pointing me there!
>I only open port 2049/tcp and everything works.

Great! If you want the full NFSv4 benefit, you'll also need to enable
delegation callbacks, which requires a TCP port in the other direction.
This port is chosen by the client, but the server makes the connection,
so you'll need to configure it in both the client and the firewall.

The port is set by an NFS sysctl parameter named "nfs_callback_tcpport",
which by default is 0 (any). You'll need to set it to some value with

	sysctl -w fs.nfs.nfs_callback_tcpport = <value>

and also in your firewall from the server->client. The range is 0 to 65535,
you can choose any convenient unused value (e.g. 2050).

BTW when NFSv4.1 is in use, this reverse connection will no longer occur.
But that's a ways off in the distribution kernels.

>However, I still see blocked connections on the firewall, coming from
>the NFS client to the NFS server: 
>...PROTO=TCP SPT=55598 DPT=111...
>rpcinfo tells me the portmapper is running at port 111 (udp and tcp).
>
>I didn't find a clear statement when googling if that should happen
>with NFSv4 or not. It doesn't seem to block the NFS share in any way,
>at least as far as I can see. 

It could be any number of things, but probably harmless and properly
ignored for NFSv4. I will guess it's something to do with the SLES11
client and other daemons.

Can you capture a trace of these messages with wireshark? The
contents of the portmapper request the client is sending will tell us
exactly what services it's trying to resolve.

Alternatively you could turn on kernel portmap debug and watch the
log. I'm not sure if SLES11 has the "rpcdebug" command installed, but
if so, on the client you could try

	rpcdebug -m rpc -s bind

then watch the syslog.

>I wouldn't mind to open tcp port 111 to the NFS server. I'm just curios
>if that behaviour is correct or not with NFSv4.

It's not part of NFSv4. But it could be a side effect of the client NFS
implementation, which also supports v2 and v3 plus NSM, NLM, etc etc.

Tom.


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment. 
http://p.sf.net/sfu/businessobjects
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Frank Steiner]

Tom Talpey wrote

> Great! If you want the full NFSv4 benefit, you'll also need to enable
> delegation callbacks, which requires a TCP port in the other direction.
> This port is chosen by the client, but the server makes the connection,
> so you'll need to configure it in both the client and the firewall.
> 
> The port is set by an NFS sysctl parameter named "nfs_callback_tcpport",
> which by default is 0 (any). You'll need to set it to some value with
> 
> 	sysctl -w fs.nfs.nfs_callback_tcpport = <value>
> 
> and also in your firewall from the server->client. The range is 0 to 65535,
> you can choose any convenient unused value (e.g. 2050).

Your mails really help :-) I've read many notes about that problem and that
the  firewall must be opened etc., but none stated how to fix those ports!

> It could be any number of things, but probably harmless and properly
> ignored for NFSv4. I will guess it's something to do with the SLES11
> client and other daemons.

Ok, good to know.

> 
> Can you capture a trace of these messages with wireshark? The
> contents of the portmapper request the client is sending will tell us
> exactly what services it's trying to resolve.
> 
> Alternatively you could turn on kernel portmap debug and watch the
> log. I'm not sure if SLES11 has the "rpcdebug" command installed, but
> if so, on the client you could try
> 
> 	rpcdebug -m rpc -s bind
> 
> then watch the syslog.

The server and clients in charge are still running SLES 10 SP2, and it
doesn't have the rpcdebug command. However, SLES 11 has it, so I will
check when all the servers and clients have been upgraded.

So far, thanks a lot again :-)
cu,
Frank

-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment. 
http://p.sf.net/sfu/businessobjects
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Tom Talpey]

At 05:18 AM 5/18/2009, Frank Steiner wrote:
>Tom Talpey wrote
...
>> Alternatively you could turn on kernel portmap debug and watch the
>> log. I'm not sure if SLES11 has the "rpcdebug" command installed, but
>> if so, on the client you could try
>> 
>> 	rpcdebug -m rpc -s bind
>> 
>> then watch the syslog.
>
>The server and clients in charge are still running SLES 10 SP2, and it
>doesn't have the rpcdebug command. However, SLES 11 has it, so I will
>check when all the servers and clients have been upgraded.

If the rpcdebug command isn't available, you can alternatively set
set the RPCDBG_BIND bit with

	echo 32 >/proc/sys/sunrpc/rpc_debug

and clear it by echoing 0. Many other bits are accessible btw, defined
in the kernel include/sunrpc/debug.h file.

Tom.


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables 
unlimited royalty-free distribution of the report engine 
for externally facing server and web deployment. 
http://p.sf.net/sfu/businessobjects
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Frank Steiner]

Tom Talpey wrote

> If the rpcdebug command isn't available, you can alternatively set
> set the RPCDBG_BIND bit with
> 
> 	echo 32 >/proc/sys/sunrpc/rpc_debug
> 
> and clear it by echoing 0. Many other bits are accessible btw, defined
> in the kernel include/sunrpc/debug.h file.

Thanks :-) Looking at the debug output I found it was a different
program but NFS that was trying to contact the portmapper :-)

cu,
Frank

-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, & 
iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Trond Myklebust]

On Thu, 2009-05-07 at 14:57 +0200, Frank Steiner wrote:
> Hi,
> 
> I'm fighting with my firewall to get nfs-over-tcp through. 
> 
> The server is outside the firewall, the client is inside. The firewall
> allows all tcp back-connections (without syn), no UDPs. Mount on the
> client side worked fine with kernel 2.6.16 in SLES 10.
> 
> Now when the NFS client is running SLES 11 with its kernel 2.6.27, 
> the NFS server tries to make UDP connections from its ports 111 and 
> 700 to different ports on the client.
> 
> If the client is running SLES 10 with 2.6.16, those connections are
> not tried from the server, no matter if the server runs 2.6.16 or
> 2.6.27.
> 
> So I've two questions:
> 1) Should nfs-over-tcp still use any UDP ports at all?
> 2) What has been changed in the client code between 2.6.16 and 2.6.27
>    that could cause this behaviour?
> 
> Is there a way to prevent those UDP connects?

The default behaviour is to always try to use UDP to talk to mountd and
the portmapper in order to minimize the number of ports that get left in
the TIME_WAIT state. If you only want to use TCP, then you might try
using '-omountproto=tcp'

Cheers
  Trond

PS: Note that nfs@lists.sourceforge.net is deprecated due to poor
anti-spam filtering. You should rather send posts directly to
linux-nfs@vger.kernel.org



------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] nfs-over-tcp still needs udp ports? (SLES 11)

[Peter Åstrand]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 749 bytes --]

On Thu, 7 May 2009, Trond Myklebust wrote:

> PS: Note that nfs@lists.sourceforge.net is deprecated due to poor
> anti-spam filtering. You should rather send posts directly to
> linux-nfs@vger.kernel.org

Note that if you are trying to subscribe to nfs@lists.sourceforge.net, 
you'll get this message:

"This list is closed to new subscriptions.  Please subscribe to
linux-nfs-u79uwXL29TaiAVqoAR/hOA@public.gmane.org  Send mail to majorodomo@vger.kernel.org to
subscribe."

majorodomo@vger.kernel.org is misspelled; should be 
majordomo-u79uwXL29TaiAVqoAR/hOA@public.gmane.org

Best regards, 
---
Peter Åstrand		ThinLinc Chief Developer
Cendio AB		http://www.cendio.com
Wallenbergs gata 4
583 30 Linköping	Phone: +46-13-21 46 00

[-- Attachment #2: Type: text/plain, Size: 432 bytes --]

------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image 
processing features enabled. http://p.sf.net/sfu/kodak-com

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs