Improving unfsd scalability

Improving unfsd scalability

[Murali Bashyam]

[-- Attachment #1.1: Type: text/plain, Size: 212 bytes --]

Hi

Has anyone attempted to speedup concurrently handling many client NFS
requests in unfsd? Any experience regarding multi-threading or running
multiple processes within unfsd? Any do's and don'ts?

Thx,
Murali

[-- Attachment #1.2: Type: text/html, Size: 232 bytes --]

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability

[Peter Astrand]

[-- Attachment #1: Type: TEXT/Plain, Size: 651 bytes --]

On Wed, 29 Nov 2006, Murali Bashyam wrote:

> Has anyone attempted to speedup concurrently handling many client NFS
> requests in unfsd? Any experience regarding multi-threading or running
> multiple processes within unfsd? Any do's and don'ts?

This is an interesting topic. I guess it depends a lot on which RPC 
library you are using. The "normal" ONC RPC implementation, found in glibc 
for example, is as far as I know not thread safe. This might be a problem. 
Correct me if I'm wrong.

Regards, 
-- 
Peter Åstrand		ThinLinc Chief Developer
Cendio AB		http://www.cendio.se
Teknikringen 3
583 30 Linköping	Phone: +46-13-21 46 00

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability

[Olaf Kirch]

On Thu, Nov 30, 2006 at 11:00:25AM +0100, Peter Astrand wrote:
> > Has anyone attempted to speedup concurrently handling many client NFS
> > requests in unfsd? Any experience regarding multi-threading or running
> > multiple processes within unfsd? Any do's and don'ts?
> 
> This is an interesting topic. I guess it depends a lot on which RPC 
> library you are using. The "normal" ONC RPC implementation, found in glibc 
> for example, is as far as I know not thread safe. This might be a problem. 
> Correct me if I'm wrong.

unfsd was able to run several process in parallel since I think 1998. Note
it's multi-process not multithread, so it's a tad inefficient - but I
haven't heard of any problems in that area.  unfsd was running on some
machines in the Suse R&D network as late as a few months ago, with 8
processes in parallel.

Olaf
-- 
Walks like a duck. Quacks like a duck. Must be a chicken.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability

[Murali Bashyam]

[-- Attachment #1.1: Type: text/plain, Size: 1662 bytes --]

Olaf

So this version of unfsd would fork several processes and there would be
some dispatching process to hand RPC requests to them? I assume these are
not multiple daemons invoked from the shell. The unfsd version available for
download seems to be strictly single process, and doesn't seem to be doing
this.

And what happens if multiple processes handle writes on behalf of different
clients to the same file? I just want to make sure the 8 process server can
handle what ever the clients do i.e lock, read, write etc. Presumably there
are inefficiencies, of course such as independent file handle caches within
each process and not much sharing between them.

Murali

On 11/30/06, Olaf Kirch <okir@suse.de> wrote:
>
> On Thu, Nov 30, 2006 at 11:00:25AM +0100, Peter Astrand wrote:
> > > Has anyone attempted to speedup concurrently handling many client NFS
> > > requests in unfsd? Any experience regarding multi-threading or running
> > > multiple processes within unfsd? Any do's and don'ts?
> >
> > This is an interesting topic. I guess it depends a lot on which RPC
> > library you are using. The "normal" ONC RPC implementation, found in
> glibc
> > for example, is as far as I know not thread safe. This might be a
> problem.
> > Correct me if I'm wrong.
>
> unfsd was able to run several process in parallel since I think 1998. Note
> it's multi-process not multithread, so it's a tad inefficient - but I
> haven't heard of any problems in that area.  unfsd was running on some
> machines in the Suse R&D network as late as a few months ago, with 8
> processes in parallel.
>
> Olaf
> --
> Walks like a duck. Quacks like a duck. Must be a chicken.
>

[-- Attachment #1.2: Type: text/html, Size: 2025 bytes --]

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability

[Olaf Kirch]

On Thu, Nov 30, 2006 at 09:58:00AM -0800, Murali Bashyam wrote:
> So this version of unfsd would fork several processes and there would be
> some dispatching process to hand RPC requests to them? I assume these are
> not multiple daemons invoked from the shell. The unfsd version available for
> download seems to be strictly single process, and doesn't seem to be doing
> this.

Search for older suse nfs-server RPMs; that unfsd should have the
support.

And no, there's no dispatching. You start "rpc.nfsd 8" and that
will open a UDP and TCP socket on port 2049, then fork 7 children.
UDP requests are handled by whichever daemon wakes up first.
Same for incoming TCP connects; after the connection is established,
it is handled by one daemon.

> And what happens if multiple processes handle writes on behalf of different
> clients to the same file? I just want to make sure the 8 process server can
> handle what ever the clients do i.e lock, read, write etc. Presumably there
> are inefficiencies, of course such as independent file handle caches within
> each process and not much sharing between them.

unfsd doesn't do NFS file locking, but otherwise the file handle cache
is robust enough to handle even writes.

Anything multithreaded would require writing your own librpc.
There's no way you can make the current RPC code reasonably
efficient wrt multithreading.

I just wish people would leave this old code to Rot In Peace :)

Olaf
-- 
Walks like a duck. Quacks like a duck. Must be a chicken.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability

[Peter Astrand]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 932 bytes --]

On Thu, 30 Nov 2006, Olaf Kirch wrote:

> Anything multithreaded would require writing your own librpc.
> There's no way you can make the current RPC code reasonably
> efficient wrt multithreading.
> 
> I just wish people would leave this old code to Rot In Peace :)

Yes, the old ONC RPC code is far from optimal. But, what do you suggest 
instead? TI-RPC? It has thread support, but it's also much more 
complicated. The implementation is not as tested and the libtirpc 
implementation is not based on the latest Sun release. And, libtirpc has 
not been ported to Windows yet, and the thread stuff makes porting 
somewhat difficult. 

Btw, I'm also not convinced that threading is the best solution. I tend to 
like multi-process or select-based solutions better.

Regards, 
-- 
Peter Åstrand		ThinLinc Chief Developer
Cendio AB		http://www.cendio.se
Teknikringen 3
583 30 Linköping	Phone: +46-13-21 46 00

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability

[Olaf Kirch]

On Thu, Nov 30, 2006 at 07:27:15PM +0100, Peter Astrand wrote:
> > I just wish people would leave this old code to Rot In Peace :)
> 
> Yes, the old ONC RPC code is far from optimal. But, what do you suggest 

Oh, I'm sorry. I was referring to unfsd.

> instead? TI-RPC? It has thread support, but it's also much more 
> complicated. The implementation is not as tested and the libtirpc 
> implementation is not based on the latest Sun release. And, libtirpc has 
> not been ported to Windows yet, and the thread stuff makes porting 
> somewhat difficult. 

I'm not a huge fan of TIRPC either.

> Btw, I'm also not convinced that threading is the best solution. I tend to 
> like multi-process or select-based solutions better.

I agree.  It makes for trickier code though.

The premise of the "java way" of writing network services (one thread
per connection) is you trade complexity for computing resources. If
you create complex applications, you need expensive programmers and
QA. If you solve a problem by throwing more threads at it, the customer
will need more expensive hardware - which is too bad, but at least
it doesn't show up on your R&D budget :)

Olaf
-- 
Walks like a duck. Quacks like a duck. Must be a chicken.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability: TI-RPC

[Tony Reix]

Le jeudi 30 novembre 2006 =E0 21:01 +0100, Olaf Kirch a =E9crit :
> =

> > instead? TI-RPC? It has thread support, but it's also much more =

> > complicated. The implementation is not as tested and the libtirpc =

> > implementation is not based on the latest Sun release. And, libtirpc ha=
s =

> > not been ported to Windows yet, and the thread stuff makes porting =

> > somewhat difficult. =

> =

> I'm not a huge fan of TIRPC either.

Hi Olaf, Peter,

We'll have a student, Cyril, in 2007 during 5 months about TI-RPC.
His main goal deals with writing tests for RPC and TI-RPC lib.
But any ideas are welcome to improve TI-RPC and make you a fan of it !
Do you have a list of complaints about TI-RPC ?

Regards,

Tony


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDE=
VDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability: TI-RPC

[Peter Astrand]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1106 bytes --]

On Fri, 1 Dec 2006, Tony Reix wrote:

> > > instead? TI-RPC? It has thread support, but it's also much more 
> > > complicated. The implementation is not as tested and the libtirpc 
> > > implementation is not based on the latest Sun release. And, libtirpc has 
> > > not been ported to Windows yet, and the thread stuff makes porting 
> > > somewhat difficult. 
> > 
> > I'm not a huge fan of TIRPC either.
> 
> Hi Olaf, Peter,
> 
> We'll have a student, Cyril, in 2007 during 5 months about TI-RPC.
> His main goal deals with writing tests for RPC and TI-RPC lib.
> But any ideas are welcome to improve TI-RPC and make you a fan of it !
> Do you have a list of complaints about TI-RPC ?

This sounds great. Have you decided on the test framework and/or language 
yet? pynfs might be useful... :-)

To all: I've created a Sourceforge project for libtirpc. No source code 
has been imported yet, but I will fix this in a near future. 

Regards, 
-- 
Peter Åstrand		ThinLinc Chief Developer
Cendio AB		http://www.cendio.se
Teknikringen 3
583 30 Linköping	Phone: +46-13-21 46 00

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability: TI-RPC test suite

[Tony Reix]

Le vendredi 01 d=E9cembre 2006 =E0 14:52 +0100, Peter Astrand a =E9crit :
> On Fri, 1 Dec 2006, Tony Reix wrote:

> > We'll have a student, Cyril, in 2007 during 5 months about TI-RPC.
> > His main goal deals with writing tests for RPC and TI-RPC lib.
> =

> This sounds great. Have you decided on the test framework and/or language =

> yet? pynfs might be useful... :-)

Not decided yet. The decision will be done after the student has built
the design of the tests to be done. C and Python are candidates. Since
the test suite is aimed to be delivered to the LTP project, we'll follow
their rules.

Though the TI-RPC already appears in FC6 with no known issues, providing
a RPC test suite will help to stabilize it.

Regards,

Tony


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDE=
VDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability: TI-RPC

[Steve Dickson]

Tony Reix wrote:
> Le jeudi 30 novembre 2006 =E0 21:01 +0100, Olaf Kirch a =E9crit :
>>> instead? TI-RPC? It has thread support, but it's also much more =

>>> complicated. The implementation is not as tested and the libtirpc =

>>> implementation is not based on the latest Sun release. And, libtirpc ha=
s =

>>> not been ported to Windows yet, and the thread stuff makes porting =

>>> somewhat difficult. =

>> I'm not a huge fan of TIRPC either.
> =

> Hi Olaf, Peter,
> =

> We'll have a student, Cyril, in 2007 during 5 months about TI-RPC.
> His main goal deals with writing tests for RPC and TI-RPC lib.
> But any ideas are welcome to improve TI-RPC and make you a fan of it !
Note... the libtirpc Tony is talking about is available in FC6...
its in the core repo... There are already a few patches that
fixed a couple of issued... but over all it seem to be working...

steved.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDE=
VDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Improving unfsd scalability

[Murali Bashyam]

[-- Attachment #1.1: Type: text/plain, Size: 936 bytes --]

Peter

The RPC implementation is not thread safe. And neither is the unfsd code
itself, there are unprotected globals such as the file handle cache etc.

Murali

On 11/30/06, Peter Astrand <astrand@cendio.se> wrote:
>
> On Wed, 29 Nov 2006, Murali Bashyam wrote:
>
> > Has anyone attempted to speedup concurrently handling many client NFS
> > requests in unfsd? Any experience regarding multi-threading or running
> > multiple processes within unfsd? Any do's and don'ts?
>
> This is an interesting topic. I guess it depends a lot on which RPC
> library you are using. The "normal" ONC RPC implementation, found in glibc
> for example, is as far as I know not thread safe. This might be a problem.
> Correct me if I'm wrong.
>
> Regards,
> --
> Peter Åstrand           ThinLinc Chief Developer
> Cendio AB               http://www.cendio.se
> Teknikringen 3
> 583 30 Linköping        Phone: +46-13-21 46 00
>

[-- Attachment #1.2: Type: text/html, Size: 1435 bytes --]

[-- Attachment #2: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs