Re: Linux NFS client over TCP

Re: Linux NFS client over TCP

[Chang Li]

> --- Trond Myklebust <trond.myklebust@fys.uio.no>
> > Subject: Re: [NFS] Linux NFS client over TCP
> > 
> > >>>>> " " == Suresh Kodati <kodatisu@yahoo.com>
> > writes:
> > 
> >      > I would like to know if this is an expected
> > behavior from NFS
> >      > client side?.
> > 
> It is under Linux, yes. I probably will get round to
finishing the TCP
> active close stuff at some point, but it is not top
> of my list of
> priorities...
> 
> Cheers,
>

Trond,
I would like to know some more details.
How does nfs client know that the status has gone to
CLOSE_WAIT state?. 
NFS client does not seem to bother about the
connection status while transacting. I thought it is
rpc that takes care of reconnection.

Please let me know How we can implement this. ( I'll
give a try ).

Chang_li

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com


-------------------------------------------------------
This SF.net email is sponsored by OSDN developer relations
Here's your chance to show off your extensive product knowledge
We want to know what you know. Tell us and you have a chance to win $100
http://www.zoomerang.com/survey.zgi?HRPT1X3RYQNC5V4MLNSV3E54
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Linux NFS client over TCP

[Trond Myklebust]

>>>>> " " == Chang Li <changli_99@yahoo.com> writes:

     > Trond, I would like to know some more details.  How does nfs
     > client know that the status has gone to CLOSE_WAIT state?. NFS
     > client does not seem to bother about the connection status
     > while transacting. I thought it is rpc that takes care of
     > reconnection.

Absolutely. NFS has no business poking its nose in the details of the
RPC layer.

     > Please let me know How we can implement this. ( I'll give a try
     > ).

Actually, I thought of an easy implementation just 30 minutes after
firing off my reply to you. Together with Chuck Lever, I've refined
the idea, but we haven't yet converted it into a patch:

Cheers,
  Trond

--------------------------------------------
Add the following fields to struct rpc_xprt:
  struct tq_struct   task_cleanup;      /* struct work_struct in 2.6.x */
  struct timer_list  timer;             /* Socket timeout counter */
  unsigned long      last_used;
 
#define XPRT_LOCKED 1
#define XPRT_SOCKET_TIMEOUT (300 * HZ)
 
In xprt_setup()
  INIT_TQUEUE(&xprt->task_cleanup, xprt_socket_autoclose, xprt);
  init_timer(&xprt->timer);
  xprt->timer.function = xprt_init_autodisconnect;
  xprt->timer.data = (unsigned long) xprt;
  last_used = jiffies;

In xprt_release()
  spin_lock_bh(&xprt->sock_lock);
  if (list_empty(&xprt->recv) && !xprt->shutdown) {
        last_used = jiffies;
        mod_timer(&xprt->timer, jiffies + XPRT_SOCKET_TIMEOUT);
  }
  spin_unlock_bh(&xprt->sock_lock);

In xprt_shutdown()
  del_timer_sync(&xprt->timer);

At the top of __xprt_lock_write(), __xprt_lock_write_next
  if (test_and_set_bit(XPRT_LOCKED, &xprt_sockstate))
    return 0;

In __xprt_release_write()
  if (xprt->snd_task == task)
    xprt->snd_task = NULL;
  smp_mb__before_clear_bit();
  clear_bit(XPRT_LOCKED, &xprt->sockstate);
  smp_mb__after_clear_bit();
  __xprt_lock_write_next(xprt);


New function:

void xprt_init_autodisconnect(unsigned long data)
{
   struct rpc_xprt *xprt = (struct rpc_xprt *)data;
   struct socket *sock = xprt->sock;

   spin_lock(&xprt->sock_lock);
   if (last_used < 200*HZ)
     goto out_abort;
   if (test_and_set_bit(XPRT_LOCKED, &xprt_sockstate))
     goto out_abort;
   spin_unlock(&xprt->sock_lock);
   /* Let keventd close the socket */
   schedule_task(&xprt->task_cleanup);
   return;
out_abort:
   spin_unlock(&xprt->sock_lock);
}

void xprt_socket_autoclose(void *args)
{
   struct rpc_xprt *xprt = (struct socket *)args;

   xprt_close(xprt);
   xprt_release_write(xprt, NULL);
}


-------------------------------------------------------
This SF.net email is sponsored by: The SF.net Donation Program.
Do you like what SourceForge.net is doing for the Open
Source Community?  Make a contribution, and help us add new
features and functionality. Click here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Linux NFS client over TCP

[Gregory Baker]

Trond,

I notice that many NFS fixes that we applied outside of the stock kernel 
(2.4.21 series) have made it into 2.4.23-rc1.  Great work!  Thanks!

Do you see any other patches you're going to try to feed to Marcello for 
2.4.23 release?

Can you comment on some of the patches on your web page that would may 
improve NFS client reliability in addition to the stock 2.4.23?

linux-2.4.23-03-soft2.dif:
linux-2.4.23-04-fix_unlink.dif:
linux-2.4.23-05-seekdir.dif:
	.
	.
	.


How "comfortable" do you feel of the value added for this patch?

linux-2.4.23-10-sock_disconnect.dif:

In an environment with 1000's of linux NFS clients and multiple NetApp 
filers? (Chuck?)

I'm trying to get a feel from those I respect most what the best 
recommended route is to produce the most reliable NFS client (for our 
environment and typical activity) based on the upcoming 2.4.23 release.

Thanks,

--Greg




Trond Myklebust wrote:
>>>>>>" " == Chang Li <changli_99@yahoo.com> writes:
> 
> 
>      > Trond, I would like to know some more details.  How does nfs
>      > client know that the status has gone to CLOSE_WAIT state?. NFS
>      > client does not seem to bother about the connection status
>      > while transacting. I thought it is rpc that takes care of
>      > reconnection.
> 
> Absolutely. NFS has no business poking its nose in the details of the
> RPC layer.
> 
>      > Please let me know How we can implement this. ( I'll give a try
>      > ).
> 
> Actually, I thought of an easy implementation just 30 minutes after
> firing off my reply to you. Together with Chuck Lever, I've refined
> the idea, but we haven't yet converted it into a patch:
> 
> Cheers,
>   Trond
> 
> --------------------------------------------
> Add the following fields to struct rpc_xprt:
>   struct tq_struct   task_cleanup;      /* struct work_struct in 2.6.x */
>   struct timer_list  timer;             /* Socket timeout counter */
>   unsigned long      last_used;
>  
> #define XPRT_LOCKED 1
> #define XPRT_SOCKET_TIMEOUT (300 * HZ)
>  
> In xprt_setup()
>   INIT_TQUEUE(&xprt->task_cleanup, xprt_socket_autoclose, xprt);
>   init_timer(&xprt->timer);
>   xprt->timer.function = xprt_init_autodisconnect;
>   xprt->timer.data = (unsigned long) xprt;
>   last_used = jiffies;
> 
> In xprt_release()
>   spin_lock_bh(&xprt->sock_lock);
>   if (list_empty(&xprt->recv) && !xprt->shutdown) {
>         last_used = jiffies;
>         mod_timer(&xprt->timer, jiffies + XPRT_SOCKET_TIMEOUT);
>   }
>   spin_unlock_bh(&xprt->sock_lock);
> 
> In xprt_shutdown()
>   del_timer_sync(&xprt->timer);
> 
> At the top of __xprt_lock_write(), __xprt_lock_write_next
>   if (test_and_set_bit(XPRT_LOCKED, &xprt_sockstate))
>     return 0;
> 
> In __xprt_release_write()
>   if (xprt->snd_task == task)
>     xprt->snd_task = NULL;
>   smp_mb__before_clear_bit();
>   clear_bit(XPRT_LOCKED, &xprt->sockstate);
>   smp_mb__after_clear_bit();
>   __xprt_lock_write_next(xprt);
> 
> 
> New function:
> 
> void xprt_init_autodisconnect(unsigned long data)
> {
>    struct rpc_xprt *xprt = (struct rpc_xprt *)data;
>    struct socket *sock = xprt->sock;
> 
>    spin_lock(&xprt->sock_lock);
>    if (last_used < 200*HZ)
>      goto out_abort;
>    if (test_and_set_bit(XPRT_LOCKED, &xprt_sockstate))
>      goto out_abort;
>    spin_unlock(&xprt->sock_lock);
>    /* Let keventd close the socket */
>    schedule_task(&xprt->task_cleanup);
>    return;
> out_abort:
>    spin_unlock(&xprt->sock_lock);
> }
> 
> void xprt_socket_autoclose(void *args)
> {
>    struct rpc_xprt *xprt = (struct socket *)args;
> 
>    xprt_close(xprt);
>    xprt_release_write(xprt, NULL);
> }
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: The SF.net Donation Program.
> Do you like what SourceForge.net is doing for the Open
> Source Community?  Make a contribution, and help us add new
> features and functionality. Click here: http://sourceforge.net/donate/
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs

-- 
----------------------------------------------------------------------
Greg Baker                                         512-602-3287 (work)
gregory.baker@amd.com                              512-603-1837 (page)
5900 E. Ben White Blvd MS 625                      512-602-6970 (fax)
Austin, TX 78741                                   512-555-1212 (info)




-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Linux NFS client over TCP

[Suresh Kodati]

Hello,
I have an NFS dir over TCP from a AIX. After 6 minutes
( ~380 secs) of idle time over the connection the
socket gets a state change, the client goes into
CLOSE_WAIT and the AIX server side goes into
FIN_WAIT_2 and eventually gets closed after
tcp_finwait2.

After the connection goes to CLOSE_WAIT from Linux
side, If I try to do some I/O over the exported
directory, the connection gets re-connected with new
port and the older connection gets teared off. ( It
seems the NFS client is trying to actually close() the
socket at the next transaction instead of closing it
on receiving FIN from the other side.).
I would like to know if this is an expected behavior
from  NFS client side?. 
Thanks 
-Suresh


__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Linux NFS client over TCP

[Trond Myklebust]

>>>>> " " == Suresh Kodati <kodatisu@yahoo.com> writes:

     > I would like to know if this is an expected behavior from NFS
     > client side?.

It is under Linux, yes. I probably will get round to finishing the TCP
active close stuff at some point, but it is not top of my list of
priorities...

Cheers,
  Trond


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs