fsid question

fsid question

[Jason Keltz]

Hi.

I'm running a RedHat Enterprise 4 system.

I have a question about the fsid option in /etc/exports that I was 
hoping someone might be able to help with.

I am working with LVM under Linux.  If I export LVM volumes via NFS, the 
fsid is based on major/minor number.  If I then remove a logical volume 
on the NFS server and reboot the server, the major and minor numbers for 
the other volumes are adjusted, which of course causes problems for the 
existing NFS exports (which either end up stale/pointing to a different 
logical volume)!   The fsid option would fix this by allowing me to 
specify a number independent of major/minor number.

A couple of questions:

1) How do I find out the fsid of an existing export so that I can 
"hard-code" it in the /etc/exports file?
2) For filesystems that are not mounted, how do I choose a number that 
the system will not try to allocate to a different nfs export that isn't 
using the fsid option?  Will the system check existing fsids, or it just 
allocated the fsid based on a formula?
3) What is the best way for me to generate my own fsid?

Any assistance would be appreciated...

Jason.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Wendy Cheng]

Jason Keltz wrote:
> Hi.
>
> I'm running a RedHat Enterprise 4 system.
>
> I have a question about the fsid option in /etc/exports that I was 
> hoping someone might be able to help with.
>
> I am working with LVM under Linux.  If I export LVM volumes via NFS, the 
> fsid is based on major/minor number.  
This statement needs correction. An NFS fsid is not based on major/minor 
numbers - it is is a 32 bit number that admin (root) can arbitrary 
choose and use to uniquely identify an NFS export. More on this in the 
following comment.
> If I then remove a logical volume 
> on the NFS server and reboot the server, the major and minor numbers for 
> the other volumes are adjusted, which of course causes problems for the 
> existing NFS exports (which either end up stale/pointing to a different 
> logical volume)!   The fsid option would fix this by allowing me to 
> specify a number independent of major/minor number.
>   
This is correct (fsid option would allow admin to identify an NFS 
export, independent of major/minor number).
> A couple of questions:
>
> 1) How do I find out the fsid of an existing export so that I can 
> "hard-code" it in the /etc/exports file?
>   
The admin (root, or whoever has access rights to /etc/exports file) can 
pick one number between 0 and 0xffffffff and add it into /etc/exports. 
It comes and goes with each "exportfs" command. This id is used to 
construct NFS filehandle that will be sent to NFS client upon requests. 
NFS client uses file handle to communicate with NFS server on which file 
they want to have access. The fsid, if specified, is part of the file 
handle that uniquely identifies an export (one entry in /etc/exports file).

So as rule of thumb, you would not want to change fsid as long as there 
are NFS clients still out there trying to access the files on the 
server. It is, however, not a permanent ID since there is no filesystem 
"on-disk" record of it.
> 2) For filesystems that are not mounted, how do I choose a number that 
> the system will not try to allocate to a different nfs export that isn't 
> using the fsid option?  Will the system check existing fsids, or it just 
> allocated the fsid based on a formula?
>   
No, NFS fsid is *not* allocated (nor generated) by system. There is no 
co-relation between the physical filesystem and NFS fsid, if the 
filesystem is not NFS exported. Note that a filesystem can be mounted 
but not NFS exported - in this case, the filesystem will not recognize 
the NFS fsid. I suspect that you confuse fsid with either filehandle or 
some specific filesystem implementation fsid (few cluster filesystems I 
know of have its own fsid - but it differs from the "NFS" fsid we 
discuss here).
> 3) What is the best way for me to generate my own fsid?
>   
It is a number picked by admin and must be between 0 and 0xFFFFFFFF.

Hopefully this makes things clear (since our cluster suites use NFS fsid 
extensively).

-- Wendy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Jason Keltz]

Hi Wendy,

Thanks for your response..

A few more comments...

On 03/26/07 17:00, Wendy Cheng wrote:
> Jason Keltz wrote:
>> 1) How do I find out the fsid of an existing export so that I can 
>> "hard-code" it in the /etc/exports file?
>>   
> The admin (root, or whoever has access rights to /etc/exports file) can 
> pick one number between 0 and 0xffffffff and add it into /etc/exports. 
> It comes and goes with each "exportfs" command. This id is used to 
> construct NFS filehandle that will be sent to NFS client upon requests. 
> NFS client uses file handle to communicate with NFS server on which file 
> they want to have access. The fsid, if specified, is part of the file 
> handle that uniquely identifies an export (one entry in /etc/exports file).
> 
> So as rule of thumb, you would not want to change fsid as long as there 
> are NFS clients still out there trying to access the files on the 
> server. It is, however, not a permanent ID since there is no filesystem 
> "on-disk" record of it.

The problem is that right now, I have several hundred clients who have 
mounted the exported filesystems via NFS where I did not specify an 
explicit fsid in the NFS sever /etc/exports file.   In order to start 
using the fsid option, I need to know the existing fsids so that I can 
hard-code those into /etc/exports.  This way, if the server is rebooted, 
the clients will not be affected.  I could then choose fsids for NEW nfs 
exported filesystems and hard-code those fsids into /etc/exports.  Is 
there a Linux command, or an entry in /proc on the file server that will 
let me know the existing fsids?

>> 3) What is the best way for me to generate my own fsid?
>>   
> It is a number picked by admin and must be between 0 and 0xFFFFFFFF.
> 

Is the fsid specified in hex?

Since the system will allocate its own fsid for NFS exported filesystems 
that do not have a user-specified fsid, how do I choose fsids for my 
user-specified fsid exports that will not conflict with those that the 
system will use?

Thanks,

Jason.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[James Pearson]

Jason Keltz wrote:
> Hi Wendy,
> 
> Thanks for your response..
> 
> A few more comments...
> 
> On 03/26/07 17:00, Wendy Cheng wrote:
> 
>>Jason Keltz wrote:
>>
>>>1) How do I find out the fsid of an existing export so that I can 
>>>"hard-code" it in the /etc/exports file?
>>>  
>>
>>The admin (root, or whoever has access rights to /etc/exports file) can 
>>pick one number between 0 and 0xffffffff and add it into /etc/exports. 
>>It comes and goes with each "exportfs" command. This id is used to 
>>construct NFS filehandle that will be sent to NFS client upon requests. 
>>NFS client uses file handle to communicate with NFS server on which file 
>>they want to have access. The fsid, if specified, is part of the file 
>>handle that uniquely identifies an export (one entry in /etc/exports file).
>>
>>So as rule of thumb, you would not want to change fsid as long as there 
>>are NFS clients still out there trying to access the files on the 
>>server. It is, however, not a permanent ID since there is no filesystem 
>>"on-disk" record of it.
> 
> 
> The problem is that right now, I have several hundred clients who have 
> mounted the exported filesystems via NFS where I did not specify an 
> explicit fsid in the NFS sever /etc/exports file.   In order to start 
> using the fsid option, I need to know the existing fsids so that I can 
> hard-code those into /etc/exports.  This way, if the server is rebooted, 
> the clients will not be affected.  I could then choose fsids for NEW nfs 
> exported filesystems and hard-code those fsids into /etc/exports.  Is 
> there a Linux command, or an entry in /proc on the file server that will 
> let me know the existing fsids?

I believe you can add a fsid to /etc/exports and re-export the file 
systems - existing mounts will not be affected. New mounts will pick up 
the new fsid. The existing mounts will also pick up the new fsid the 
next time they remount the file system.

I've certainly done this in the past and had no problems.

James Pearson

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Jason Keltz]

On 03/27/07 08:56, James Pearson wrote:
> Jason Keltz wrote:
>> The problem is that right now, I have several hundred clients who have 
>> mounted the exported filesystems via NFS where I did not specify an 
>> explicit fsid in the NFS sever /etc/exports file.   In order to start 
>> using the fsid option, I need to know the existing fsids so that I can 
>> hard-code those into /etc/exports.  This way, if the server is 
>> rebooted, the clients will not be affected.  I could then choose fsids 
>> for NEW nfs exported filesystems and hard-code those fsids into 
>> /etc/exports.  Is there a Linux command, or an entry in /proc on the 
>> file server that will let me know the existing fsids?
> 
> I believe you can add a fsid to /etc/exports and re-export the file 
> systems - existing mounts will not be affected. New mounts will pick up 
> the new fsid. The existing mounts will also pick up the new fsid the 
> next time they remount the file system.
> 
> I've certainly done this in the past and had no problems.
> 
> James Pearson

Hi James.

Thanks for your message.

I'm not sure that I understand this though.  If the system is generating 
an fsid for NFS exports in the absence of a user-specified fsid, then 
changing the fsid and re-exporting the filesystem *should* essentially 
confuse the client? How does one choose an fsid that is not already in 
use on the system? I imagine there must be some way to list the 
currently used fsids for exported filesystems.

Jas.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[James Pearson]

Jason Keltz wrote:

> Hi James.
> 
> Thanks for your message.
> 
> I'm not sure that I understand this though.  If the system is generating 
> an fsid for NFS exports in the absence of a user-specified fsid, then 
> changing the fsid and re-exporting the filesystem *should* essentially 
> confuse the client? How does one choose an fsid that is not already in 
> use on the system? I imagine there must be some way to list the 
> currently used fsids for exported filesystems.

See the thread starting at:

<http://marc.info/?l=linux-nfs&m=110295083427898&w=2>

James Pearson


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Wendy Cheng]

Jason Keltz wrote:
> .........................................If the system is generating 
> an fsid for NFS exports in the absence of a user-specified fsid, then 
> changing the fsid and re-exporting the filesystem *should* essentially 
> confuse the client? How does one choose an fsid that is not already in 
> use on the system? I imagine there must be some way to list the 
> currently used fsids for exported filesystems.
>   

I think I see where the confusion comes from. To understand this, let's 
start with a thing called NFS "file handle".

When an NFS client request file access, server packages few info into a 
max 64 bytes identifier called "file handle" that is sent back to 
client. Depending on the version of Linux OS you have, it may consist of 
file inode number, directory inode number, export device major and minor 
number, or the admin-specified "fsid", etc. If the /etc/exports doesn't 
specify an "fsid", then the major and minor numbers of the exported 
device are used. Each file handle has a field called "fsid-type" to tell 
whether this file handle uses admin-specified "fsid" or device 
major-minor number.

In your case, the current FileHandle sent to NFS clients uses device 
major and minor numbers. You can safely pick *any* number you like as 
the fsid and re-exports the entries (say by doing an "exportfs -a" 
command). Any *new* NFS client requests will get a new FileHandle that 
uses the specified "fsid". However, the *existing" FileHandle(s) any NFS 
client still keeps and uses will have the "old" way - that is the device 
major-minor pair. The server knows how to interpret them (based on the 
type specified in the "fsid-type" field of the file handle).

This implies the following:

1. If server for whatever reason gets rebooted and somehow the device 
major-minor number gets altered, the client that keeps the "old" file 
handle will get a "stale file handle" error. Some of the NFS clients 
will re-do lookup to obtain the correct FileHandle(s). Some of them will 
simply fail. There is really not much you can do as a system administrator.
2. You want to keep the "fsid" same (and unique) for obvious reasons 
after you export them.
3. Any 32 bit integer can be used (say, 1, 2, 3, etc - no need to use 
hex number).

-- Wendy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Jason Keltz]

On 03/27/07 10:27, James Pearson wrote:
> Jason Keltz wrote:
> 
>> Hi James.
>>
>> Thanks for your message.
>>
>> I'm not sure that I understand this though.  If the system is 
>> generating an fsid for NFS exports in the absence of a user-specified 
>> fsid, then changing the fsid and re-exporting the filesystem *should* 
>> essentially confuse the client? How does one choose an fsid that is 
>> not already in use on the system? I imagine there must be some way to 
>> list the currently used fsids for exported filesystems.
> 
> See the thread starting at:
> 
> <http://marc.info/?l=linux-nfs&m=110295083427898&w=2>

That makes sense, but what is still not clear to me is why I can't have 
the system show me the existing fsid that it is using in the absence of 
a user-specified fsid.

Jason.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[J. Bruce Fields]

On Tue, Mar 27, 2007 at 10:57:34AM -0400, Jason Keltz wrote:
> That makes sense, but what is still not clear to me is why I can't have 
> the system show me the existing fsid that it is using in the absence of 
> a user-specified fsid.

There *is* no "existing fsid".  The fsid that you provide in the "fsid="
export option is totally different from what's used by default to
generate filehandles for that filesystem.

--b.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[J. Bruce Fields]

On Tue, Mar 27, 2007 at 10:52:50AM -0400, Wendy Cheng wrote:
> 1. If server for whatever reason gets rebooted and somehow the device 
> major-minor number gets altered, the client that keeps the "old" file 
> handle will get a "stale file handle" error.

Actually, have you checked this?  From a quick look at

	linux/fs/nfsd/nfsfh.c:fh_verify()
and

	nfs-utils/utils/mountd/cache.c:nfsd_fh()

it looks to me like nfsd should still allow both the old and new
filehandles.

--b.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Roger Heflin]

Wendy Cheng wrote:

> 1. If server for whatever reason gets rebooted and somehow the device 
> major-minor number gets altered, the client that keeps the "old" file 
> handle will get a "stale file handle" error. Some of the NFS clients 
> will re-do lookup to obtain the correct FileHandle(s). Some of them will 
> simply fail. There is really not much you can do as a system administrator.

I think this can also happen if you are using the major+minor numbers with
say lvm, and change the order of the underlying LVM's, this might even
be a risk with the fsid option if the underlying filesystem that it was
supposed to be exporting fails to get mounted before the export.

Server reboots, a different device is put at the same major and minor 
numbers
than was there before,now  the server now gets hammered with inode numbers
that don't exist  (valid range for filesystem, but don't exist on the "new"
filesystem that is at the same major and minor numbers as the old one was).

I have seen this actually completely use up all of the CPU on a NFS server,
eventually causing it to be completely unusable, and of course the issue 
survives
reboots of the server as the requests are coming from the clients, and the
clients don't appear to ever give up, and if you have enough clients you
can make the server unusable fairly quickly.   I have seen the machine
actually hang bringing up NFS from this.

                                           Roger

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Trond Myklebust]

On Tue, 2007-03-27 at 11:23 -0400, J. Bruce Fields wrote:
> On Tue, Mar 27, 2007 at 10:52:50AM -0400, Wendy Cheng wrote:
> > 1. If server for whatever reason gets rebooted and somehow the device 
> > major-minor number gets altered, the client that keeps the "old" file 
> > handle will get a "stale file handle" error.
> 
> Actually, have you checked this?  From a quick look at
> 
> 	linux/fs/nfsd/nfsfh.c:fh_verify()
> and
> 
> 	nfs-utils/utils/mountd/cache.c:nfsd_fh()
> 
> it looks to me like nfsd should still allow both the old and new
> filehandles.

That is correct. A server kernel upgrade should not normally result in a
client error. See also Neil's recently posted patches that replace the
fileid with the disk UUID. Those again attempt to preserve backward
compatibility for filehandles.

Cheers,
  Trond


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Chris Osicki]

On Mon, 26 Mar 2007 17:00:36 -0400
Wendy Cheng <wcheng@redhat.com> wrote:

Hi

> Jason Keltz wrote:
> > Hi.
> >
> > I'm running a RedHat Enterprise 4 system.
> >
> > I have a question about the fsid option in /etc/exports that I was 
> > hoping someone might be able to help with.
> >
> > I am working with LVM under Linux.  If I export LVM volumes via NFS, the 
> > fsid is based on major/minor number.  
> This statement needs correction. An NFS fsid is not based on major/minor 
> numbers - it is is a 32 bit number that admin (root) can arbitrary 
> choose and use to uniquely identify an NFS export. More on this in the 
> following comment.

And this statement needs correction too ;-)
fsid _is_ based on major/minor number. I learned it the hard way,
seeing two filesystem swapped on NFS clients because of minor
number change after switching a cluster package to another node.

Then I read carefully fsid section in man 5 exports.

Regards,
Chris

[snip] 

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Wendy Cheng]

Chris Osicki wrote:
> On Mon, 26 Mar 2007 17:00:36 -0400
> Wendy Cheng <wcheng@redhat.com> wrote:
>
> Hi
>
>   
>> Jason Keltz wrote:
>>     
>>> Hi.
>>>
>>> I'm running a RedHat Enterprise 4 system.
>>>
>>> I have a question about the fsid option in /etc/exports that I was 
>>> hoping someone might be able to help with.
>>>
>>> I am working with LVM under Linux.  If I export LVM volumes via NFS, the 
>>> fsid is based on major/minor number.  
>>>       
>> This statement needs correction. An NFS fsid is not based on major/minor 
>> numbers - it is is a 32 bit number that admin (root) can arbitrary 
>> choose and use to uniquely identify an NFS export. More on this in the 
>> following comment.
>>     
>
> And this statement needs correction too ;-)
> fsid _is_ based on major/minor number. I learned it the hard way,
> seeing two filesystem swapped on NFS clients because of minor
> number change after switching a cluster package to another node.
>
> Then I read carefully fsid section in man 5 exports.
>
>   
Well, I still don't see anywhere there is a documented definition that 
says "(NFS) fsid _is_ based on major/minor number". The man page 
describes it as "the filesystem identification portion of the file 
handle and file attributes used on the wire". My opinion is that an NFS 
"fsid" is the number you specify in the export file - nothing more.

On the other hand, arguments like these are kind of silly. The important 
thing is to advocate and make sure people know how to use "fsid" in a 
clustered environment.

-- Wendy


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: fsid question

[Klaus Steinberger]

[-- Attachment #1.1: Type: text/plain, Size: 1323 bytes --]


> Well, I still don't see anywhere there is a documented definition that
> says "(NFS) fsid _is_ based on major/minor number". The man page
> describes it as "the filesystem identification portion of the file

I could not tell you out of memory where it is defined, too bad it's not in 
the man page: Should be enhanced!

Definitly the default for fsid is based on major/minor number, as long as you 
don't have a fsid option in you're exports file.

> On the other hand, arguments like these are kind of silly. The important
> thing is to advocate and make sure people know how to use "fsid" in a
> clustered environment.

Not only in clustered environment. It is necessary to use fsid in an LVM 
environment. I learned this also the hard way, I always wondered about stale 
handles on clients after server reboots and reexports, until I learned that 
lvm minor numbers can change, and until I discovered fsid.

But be careful! There is nothing that prevents you from using same fsid for 
different exports. 

Sincerly,
Klaus



-- 
Klaus Steinberger         Beschleunigerlaboratorium
Phone: (+49 89)289 14287  Am Coulombwall 6, D-85748 Garching, Germany
FAX:   (+49 89)289 14280  EMail: Klaus.Steinberger@Physik.Uni-Muenchen.DE
URL: http://www.physik.uni-muenchen.de/~Klaus.Steinberger/

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2002 bytes --]

[-- Attachment #2: Type: text/plain, Size: 345 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs