NFSv4 vs NFSv3 with MPICH2-1.4

NFSv4 vs NFSv3 with MPICH2-1.4

[Gregory Magoon]

Hi,
I have been having issues when running MPICH2-1.4 on a NFSv4 filesystem (they
both seem to work fine on their own). When I mount as NFS version 4, I get lots
of NFS traffic on my internal network (seen when I run sudo iftop -i eth1) and
the mpich2 tests time out or give i/o errors. However, when I mount the NFS as
version 3, the MPICH2 tests seem to work without any major problems. Even
though I can get this working by switching to NFSv3, I would like to get MPICH2
working with NFSv4, if possible. Is there something in my NFS configuration (see
below) that is not possible with NFSv4 or is this some sort of conflict between
MPICH2-1.4 and NFSv4?

Thanks,
Greg


Here is the /etc/exports on the NFS server:
/usr/local 192.168.2.0/24(async,fsid=1,ro,no_root_squash,no_subtree_check)
/home 192.168.2.0/24(async,fsid=2,rw,no_root_squash,no_subtree_check)

Here are the NFS mounting details from "nfsstat -m":

NFSv4 (hangs and times out, with large quantity of NFS traffic):
 /home from 192.168.2.11:/home/
 Flags:
 rw,relatime,vers=4,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,port=0,timeo=20,retrans=32,sec=sys,clientaddr=192.168.2.1,minorversion=0,local_lock=none,addr=192.168.2.11

/usr/local from 192.168.2.11:/usr/local/
 Flags:
 ro,relatime,vers=4,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,port=0,timeo=20,retrans=32,sec=sys,clientaddr=192.168.2.1,minorversion=0,local_lock=none,addr=192.168.2.11

NFSv3 (works; mounted in fstab same as above except with nfs replacing nfs4 and
 nfsv and an extra option nfsvers=3):
 /home from 192.168.2.11:/home
 Flags:
 rw,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=20,retrans=32,sec=sys,mountaddr=192.168.2.11,mountvers=3,mountport=55359,mountproto=udp,local_lock=none,addr=192.168.2.11

/usr/local from 192.168.2.11:/usr/local
 Flags:
 ro,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=20,retrans=32,sec=sys,mountaddr=192.168.2.11,mountvers=3,mountport=55359,mountproto=udp,local_lock=none,addr=192.168.2.11

Re: NFSv4 vs NFSv3 with MPICH2-1.4

[Trond Myklebust]

On Thu, 2011-07-28 at 15:23 -0400, Gregory Magoon wrote: 
> Hi,
> I have been having issues when running MPICH2-1.4 on a NFSv4 filesystem (they
> both seem to work fine on their own). When I mount as NFS version 4, I get lots
> of NFS traffic on my internal network (seen when I run sudo iftop -i eth1) and
> the mpich2 tests time out or give i/o errors. However, when I mount the NFS as
> version 3, the MPICH2 tests seem to work without any major problems. Even
> though I can get this working by switching to NFSv3, I would like to get MPICH2
> working with NFSv4, if possible. Is there something in my NFS configuration (see
> below) that is not possible with NFSv4 or is this some sort of conflict between
> MPICH2-1.4 and NFSv4?
> 
> Thanks,
> Greg
> 
> 
> Here is the /etc/exports on the NFS server:
> /usr/local 192.168.2.0/24(async,fsid=1,ro,no_root_squash,no_subtree_check)
> /home 192.168.2.0/24(async,fsid=2,rw,no_root_squash,no_subtree_check)
> 
> Here are the NFS mounting details from "nfsstat -m":
> 
> NFSv4 (hangs and times out, with large quantity of NFS traffic):
>  /home from 192.168.2.11:/home/
>  Flags:
>  rw,relatime,vers=4,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,port=0,timeo=20,retrans=32,sec=sys,clientaddr=192.168.2.1,minorversion=0,local_lock=none,addr=192.168.2.11
> 

This is getting to be a FAQ...

2 second timeouts over TCP are just plain wrong: they will cause you to
continuously disconnect and reconnect (as 'netstat -t' is bound to show
you). Please don't muck with the default timeo=600.

There is also no reason to mess with rsize and wsize: allow the client
and server to autonegotiate.


-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: NFSv4 vs NFSv3 with MPICH2-1.4

[Gregory Magoon]

Thanks for the tips...unfortunately, making the changes you suggest (removing
timeo, rsize, wsize options) doesn't seem to address the issue with MPICH2 and
NFSv4.

Greg

Quoting Trond Myklebust <Trond.Myklebust@netapp.com>:
>
> This is getting to be a FAQ...
>
> 2 second timeouts over TCP are just plain wrong: they will cause you to
> continuously disconnect and reconnect (as 'netstat -t' is bound to show
> you). Please don't muck with the default timeo=600.
>
> There is also no reason to mess with rsize and wsize: allow the client
> and server to autonegotiate.
>

Re: NFSv4 vs NFSv3 with MPICH2-1.4

[Trond Myklebust]

On Thu, 2011-07-28 at 17:24 -0400, Gregory Magoon wrote: 
> Thanks for the tips...unfortunately, making the changes you suggest (removing
> timeo, rsize, wsize options) doesn't seem to address the issue with MPICH2 and
> NFSv4.

Have you turned off delegations on the server? I wouldn't expect them to
help much on an MPI workload.

Otherwise, you might want to post a comparison of your results from
'nfsstat' for your workload on NFSv3 and NFSv4.


-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: NFSv4 vs NFSv3 with MPICH2-1.4

[Gregory Magoon]

I'll try this the next time I get a chance.

Just to be clear, I don't think it is a performance issue; I think it is just
getting stuck.

Greg

Quoting Trond Myklebust <Trond.Myklebust@netapp.com>:


> Have you turned off delegations on the server? I wouldn't expect them to
> help much on an MPI workload.
>

Re: NFSv4 vs NFSv3 with MPICH2-1.4

[Boaz Harrosh]

On 07/28/2011 03:01 PM, Gregory Magoon wrote:
> I'll try this the next time I get a chance.
> 
> Just to be clear, I don't think it is a performance issue; I think it is just
> getting stuck.
> 
> Greg
> 
> Quoting Trond Myklebust <Trond.Myklebust@netapp.com>:
> 
> 
>> Have you turned off delegations on the server? I wouldn't expect them to
>> help much on an MPI workload.
>>
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Gregory

We are using MPICH2-1.3.1 and the IOR mpich test. as well as the mdtest
test. And have had no issues so far with nfsv4 nfsv4.1 and pnfs. In fact
this is our standard performance test. 

What tests are you using? 
Do you know of any major changes between MPICH2-1.3.1 and MPICH2-1.4? 
Also are you using the builtin nfs-client driver or the POSIX interface?

Boaz

Re: NFSv4 vs NFSv3 with MPICH2-1.4

[Gregory Magoon]

Unfortunately, I'm not familiar enough with MPICH2 to have an idea about
significant changes between version 1.3 and 1.4, but other evidence suggests
that the version is not the issue and that I would have the same problem with
v1.3.

I'm using the MPICH2 test suite invoked by "make testing" (see below 
for initial
output).

I'm using the nfs-kernel-server and nfs-common Ubuntu packages (natty 
release).

Thanks,
Greg

user@node01:~/Molpro/src/mpich2-1.4$ make testing
(cd test && make testing)
make[1]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test'
(NOXMLCLOSE=YES && export NOXMLCLOSE && cd mpi && make testing)
make[2]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test/mpi'
./runtests -srcdir=. -tests=testlist \
                    -mpiexec=/home/user/Molpro/src/mpich2-install/bin/mpiexec \
                    -xmlfile=summary.xml
Looking in ./testlist
Processing directory attr
Looking in ./attr/testlist
Processing directory coll
Looking in ./coll/testlist
Unexpected output in allred: [mpiexec@node01] APPLICATION TIMED OUT
Unexpected output in allred: [proxy:0:0@node01] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
Unexpected output in allred: [proxy:0:0@node01] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
Unexpected output in allred: [proxy:0:0@node01] main 
(./pm/pmiserv/pmip.c:226):
demux engine error waiting for event
Unexpected output in allred: [mpiexec@node01] HYDT_bscu_wait_for_completion
(./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated
badly; aborting
Unexpected output in allred: [mpiexec@node01] HYDT_bsci_wait_for_completion
(./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for
completion
Unexpected output in allred: [mpiexec@node01] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:189): launcher returned error waiting for
completion
Unexpected output in allred: [mpiexec@node01] main (./ui/mpich/mpiexec.c:397):
process manager error waiting for completion
Program allred exited without No Errors

>
> Hi Gregory
>
> We are using MPICH2-1.3.1 and the IOR mpich test. as well as the mdtest
> test. And have had no issues so far with nfsv4 nfsv4.1 and pnfs. In fact
> this is our standard performance test.
>
> What tests are you using?
> Do you know of any major changes between MPICH2-1.3.1 and MPICH2-1.4?
> Also are you using the builtin nfs-client driver or the POSIX interface?
>
> Boaz
>

Re: NFSv4 vs NFSv3 with MPICH2-1.4

[Boaz Harrosh]

On 07/28/2011 04:15 PM, Gregory Magoon wrote:
> Unfortunately, I'm not familiar enough with MPICH2 to have an idea about
> significant changes between version 1.3 and 1.4, but other evidence suggests
> that the version is not the issue and that I would have the same problem with
> v1.3.
> 
> I'm using the MPICH2 test suite invoked by "make testing" (see below 
> for initial
> output).
> 
> I'm using the nfs-kernel-server and nfs-common Ubuntu packages (natty 
> release).
> 

You have not answered the most important question:
>> Also are you using the builtin nfs-client driver or the POSIX interface?

Which I'll assume means you don't know. So I'll try to elaborate. Just for
background, I've never used "make tests" before all I used was IOR & mdtest.

Now if you print the usage string for IOR you get this option:

	-a S  api --  API for I/O [POSIX|MPIIO|HDF5|NCMPI]

I'm not familiar with the code but what I understand is only "-a POSIX" will actually
use the regular Kernel VFS interface for read/writing of files. The other options
have different drivers for different protocols. I do not know first hand, but I once
heard in a conference that -a MPIIO has a special NFS driver that uses better NFS
semantics and avoids the POSIX semantics which are bad for big cluster performance.
All this is speculations and rumors on my part, and you will need to consult with the
mpich guys.

Now I can imagine that a "make tests" would try all possible combinations of "-a S"
So you'll need to dig out what is the falling test and is it really using the Kernel
NFS driver at that point. (I bet if you do a tcpdump like Bruce said the guys here will
be able to see if this is a Linux NFS or not)

I CC: Bill Loewe that might know much more then me about this subject. And please do
speak with the MPICH people (But keep us in the loop it is interesting to know)

Thanks
Boaz

> Thanks,
> Greg
> 
> user@node01:~/Molpro/src/mpich2-1.4$ make testing
> (cd test && make testing)
> make[1]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test'
> (NOXMLCLOSE=YES && export NOXMLCLOSE && cd mpi && make testing)
> make[2]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test/mpi'
> ./runtests -srcdir=. -tests=testlist \
>                     -mpiexec=/home/user/Molpro/src/mpich2-install/bin/mpiexec \
>                     -xmlfile=summary.xml
> Looking in ./testlist
> Processing directory attr
> Looking in ./attr/testlist
> Processing directory coll
> Looking in ./coll/testlist
> Unexpected output in allred: [mpiexec@node01] APPLICATION TIMED OUT
> Unexpected output in allred: [proxy:0:0@node01] HYD_pmcd_pmip_control_cmd_cb
> (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
> Unexpected output in allred: [proxy:0:0@node01] HYDT_dmxu_poll_wait_for_event
> (./tools/demux/demux_poll.c:77): callback returned error status
> Unexpected output in allred: [proxy:0:0@node01] main 
> (./pm/pmiserv/pmip.c:226):
> demux engine error waiting for event
> Unexpected output in allred: [mpiexec@node01] HYDT_bscu_wait_for_completion
> (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated
> badly; aborting
> Unexpected output in allred: [mpiexec@node01] HYDT_bsci_wait_for_completion
> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for
> completion
> Unexpected output in allred: [mpiexec@node01] HYD_pmci_wait_for_completion
> (./pm/pmiserv/pmiserv_pmci.c:189): launcher returned error waiting for
> completion
> Unexpected output in allred: [mpiexec@node01] main (./ui/mpich/mpiexec.c:397):
> process manager error waiting for completion
> Program allred exited without No Errors
> 
>>
>> Hi Gregory
>>
>> We are using MPICH2-1.3.1 and the IOR mpich test. as well as the mdtest
>> test. And have had no issues so far with nfsv4 nfsv4.1 and pnfs. In fact
>> this is our standard performance test.
>>
>> What tests are you using?
>> Do you know of any major changes between MPICH2-1.3.1 and MPICH2-1.4?
>> Also are you using the builtin nfs-client driver or the POSIX interface?
>>
>> Boaz
>>
> 
> 

RE: NFSv4 vs NFSv3 with MPICH2-1.4

[Loewe, Bill]

SGkgR3JlZywNCg0KSU9SIGlzIGluZGVwZW5kZW50IG9mIE1QSUNIMiwgYnV0IGRvZXMgcmVxdWly
ZSBNUEkgZm9yIHByb2Nlc3MgY29vcmRpbmF0aW9uLiAgQnkgZGVmYXVsdCwgSU9SIHdpbGwgdXNl
IHRoZSAiLWEgUE9TSVgiIG9wdGlvbiBmb3Igc3RhbmRhcmQgUE9TSVggSS9PIC0tIG9wZW4oKSwg
d3JpdGUoKSwgY2xvc2UoKSwgZXRjLg0KDQpJbiBhZGRpdGlvbiwgSU9SIGNhbiB1c2UgdGhlIE1Q
SS1JTyBsaWJyYXJ5IGNhbGxzIChNUElfRmlsZV9vcGVuKCksIGV0Yy4pIHRvIHBlcmZvcm0gSS9P
Lg0KDQpGb3IgdGhlIGJ1aWxkIHByb2Nlc3Mgb2YgTVBJQ0gyICJtYWtlIHRlc3RzIiBleGVyY2lz
ZXMgdGhpcyBNUEktSU8gKFJPTUlPKSBpbnRlcmZhY2Ugd2hpY2ggdXNlcyBhbiBBRElPIChBYnN0
cmFjdC1EZXZpY2UgSW50ZXJmYWNlIGZvciBJL08pIGxheWVyLiAgQURJTyBjYW4gaW50ZXJmYWNl
IHRvIGRpZmZlcmVudCBmaWxlIHN5c3RlbXMgKE5GUywgUGFuRlMsIFBWRlMyLCBMdXN0cmUsIGUu
Zy4pLg0KDQpUaGUgZXJyb3JzIHlvdSdyZSBlbmNvdW50ZXJpbmcgaW4gIm1ha2UgdGVzdHMiIGZv
ciBNUElDSDIgZG8gbm90IGFwcGVhciB0byBiZSB0ZXN0aW5nIHRoZSBJL08sIGhvd2V2ZXIsIGJ1
dCBzZWVtIHRvIGJlIGFuIGlzc3VlIHdpdGggdGhlIGxhdW5jaGVyIGZvciB0aGUgdGVzdHMgaW4g
Z2VuZXJhbC4gIEkgYWdyZWUgd2l0aCBCb2F6IHRoYXQgaXQgbWF5IG1ha2Ugc2Vuc2UgdG8gZm9s
bG93IHVwIHdpdGggdGhlIE1QSUNIIGRldmVsb3BlcnMgZm9yIHRoaXMuICBVbmRlciB0aGVpciBt
YWluIHBhZ2UgKGh0dHA6Ly93d3cubWNzLmFubC5nb3YvcmVzZWFyY2gvcHJvamVjdHMvbXBpY2gy
LykgdGhleSBoYXZlIGEgc3VwcG9ydCBwdWxsZG93biB3aXRoIEZBUSBhbmQgYSBtYWlsaW5nIGxp
c3QuICBUaGV5IG1heSBiZSBhYmxlIHRvIGhlbHAgcmVzb2x2ZSB0aGlzIGZvciB5b3UuDQoNClRo
YW5rcywNCg0KLS1CaWxsLg0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogSGFy
cm9zaCwgQm9heiANClNlbnQ6IEZyaWRheSwgSnVseSAyOSwgMjAxMSA4OjIwIFBNDQpUbzogR3Jl
Z29yeSBNYWdvb24NCkNjOiBUcm9uZCBNeWtsZWJ1c3Q7IGxpbnV4LW5mc0B2Z2VyLmtlcm5lbC5v
cmc7IEouIEJydWNlIEZpZWxkczsgTG9ld2UsIEJpbGwNClN1YmplY3Q6IFJlOiBORlN2NCB2cyBO
RlN2MyB3aXRoIE1QSUNIMi0xLjQNCg0KT24gMDcvMjgvMjAxMSAwNDoxNSBQTSwgR3JlZ29yeSBN
YWdvb24gd3JvdGU6DQo+IFVuZm9ydHVuYXRlbHksIEknbSBub3QgZmFtaWxpYXIgZW5vdWdoIHdp
dGggTVBJQ0gyIHRvIGhhdmUgYW4gaWRlYSBhYm91dA0KPiBzaWduaWZpY2FudCBjaGFuZ2VzIGJl
dHdlZW4gdmVyc2lvbiAxLjMgYW5kIDEuNCwgYnV0IG90aGVyIGV2aWRlbmNlIHN1Z2dlc3RzDQo+
IHRoYXQgdGhlIHZlcnNpb24gaXMgbm90IHRoZSBpc3N1ZSBhbmQgdGhhdCBJIHdvdWxkIGhhdmUg
dGhlIHNhbWUgcHJvYmxlbSB3aXRoDQo+IHYxLjMuDQo+IA0KPiBJJ20gdXNpbmcgdGhlIE1QSUNI
MiB0ZXN0IHN1aXRlIGludm9rZWQgYnkgIm1ha2UgdGVzdGluZyIgKHNlZSBiZWxvdyANCj4gZm9y
IGluaXRpYWwNCj4gb3V0cHV0KS4NCj4gDQo+IEknbSB1c2luZyB0aGUgbmZzLWtlcm5lbC1zZXJ2
ZXIgYW5kIG5mcy1jb21tb24gVWJ1bnR1IHBhY2thZ2VzIChuYXR0eSANCj4gcmVsZWFzZSkuDQo+
IA0KDQpZb3UgaGF2ZSBub3QgYW5zd2VyZWQgdGhlIG1vc3QgaW1wb3J0YW50IHF1ZXN0aW9uOg0K
Pj4gQWxzbyBhcmUgeW91IHVzaW5nIHRoZSBidWlsdGluIG5mcy1jbGllbnQgZHJpdmVyIG9yIHRo
ZSBQT1NJWCBpbnRlcmZhY2U/DQoNCldoaWNoIEknbGwgYXNzdW1lIG1lYW5zIHlvdSBkb24ndCBr
bm93LiBTbyBJJ2xsIHRyeSB0byBlbGFib3JhdGUuIEp1c3QgZm9yDQpiYWNrZ3JvdW5kLCBJJ3Zl
IG5ldmVyIHVzZWQgIm1ha2UgdGVzdHMiIGJlZm9yZSBhbGwgSSB1c2VkIHdhcyBJT1IgJiBtZHRl
c3QuDQoNCk5vdyBpZiB5b3UgcHJpbnQgdGhlIHVzYWdlIHN0cmluZyBmb3IgSU9SIHlvdSBnZXQg
dGhpcyBvcHRpb246DQoNCgktYSBTICBhcGkgLS0gIEFQSSBmb3IgSS9PIFtQT1NJWHxNUElJT3xI
REY1fE5DTVBJXQ0KDQpJJ20gbm90IGZhbWlsaWFyIHdpdGggdGhlIGNvZGUgYnV0IHdoYXQgSSB1
bmRlcnN0YW5kIGlzIG9ubHkgIi1hIFBPU0lYIiB3aWxsIGFjdHVhbGx5DQp1c2UgdGhlIHJlZ3Vs
YXIgS2VybmVsIFZGUyBpbnRlcmZhY2UgZm9yIHJlYWQvd3JpdGluZyBvZiBmaWxlcy4gVGhlIG90
aGVyIG9wdGlvbnMNCmhhdmUgZGlmZmVyZW50IGRyaXZlcnMgZm9yIGRpZmZlcmVudCBwcm90b2Nv
bHMuIEkgZG8gbm90IGtub3cgZmlyc3QgaGFuZCwgYnV0IEkgb25jZQ0KaGVhcmQgaW4gYSBjb25m
ZXJlbmNlIHRoYXQgLWEgTVBJSU8gaGFzIGEgc3BlY2lhbCBORlMgZHJpdmVyIHRoYXQgdXNlcyBi
ZXR0ZXIgTkZTDQpzZW1hbnRpY3MgYW5kIGF2b2lkcyB0aGUgUE9TSVggc2VtYW50aWNzIHdoaWNo
IGFyZSBiYWQgZm9yIGJpZyBjbHVzdGVyIHBlcmZvcm1hbmNlLg0KQWxsIHRoaXMgaXMgc3BlY3Vs
YXRpb25zIGFuZCBydW1vcnMgb24gbXkgcGFydCwgYW5kIHlvdSB3aWxsIG5lZWQgdG8gY29uc3Vs
dCB3aXRoIHRoZQ0KbXBpY2ggZ3V5cy4NCg0KTm93IEkgY2FuIGltYWdpbmUgdGhhdCBhICJtYWtl
IHRlc3RzIiB3b3VsZCB0cnkgYWxsIHBvc3NpYmxlIGNvbWJpbmF0aW9ucyBvZiAiLWEgUyINClNv
IHlvdSdsbCBuZWVkIHRvIGRpZyBvdXQgd2hhdCBpcyB0aGUgZmFsbGluZyB0ZXN0IGFuZCBpcyBp
dCByZWFsbHkgdXNpbmcgdGhlIEtlcm5lbA0KTkZTIGRyaXZlciBhdCB0aGF0IHBvaW50LiAoSSBi
ZXQgaWYgeW91IGRvIGEgdGNwZHVtcCBsaWtlIEJydWNlIHNhaWQgdGhlIGd1eXMgaGVyZSB3aWxs
DQpiZSBhYmxlIHRvIHNlZSBpZiB0aGlzIGlzIGEgTGludXggTkZTIG9yIG5vdCkNCg0KSSBDQzog
QmlsbCBMb2V3ZSB0aGF0IG1pZ2h0IGtub3cgbXVjaCBtb3JlIHRoZW4gbWUgYWJvdXQgdGhpcyBz
dWJqZWN0LiBBbmQgcGxlYXNlIGRvDQpzcGVhayB3aXRoIHRoZSBNUElDSCBwZW9wbGUgKEJ1dCBr
ZWVwIHVzIGluIHRoZSBsb29wIGl0IGlzIGludGVyZXN0aW5nIHRvIGtub3cpDQoNClRoYW5rcw0K
Qm9heg0KDQo+IFRoYW5rcywNCj4gR3JlZw0KPiANCj4gdXNlckBub2RlMDE6fi9Nb2xwcm8vc3Jj
L21waWNoMi0xLjQkIG1ha2UgdGVzdGluZw0KPiAoY2QgdGVzdCAmJiBtYWtlIHRlc3RpbmcpDQo+
IG1ha2VbMV06IEVudGVyaW5nIGRpcmVjdG9yeSBgL2hvbWUvdXNlci9Nb2xwcm8vc3JjL21waWNo
Mi0xLjQvdGVzdCcNCj4gKE5PWE1MQ0xPU0U9WUVTICYmIGV4cG9ydCBOT1hNTENMT1NFICYmIGNk
IG1waSAmJiBtYWtlIHRlc3RpbmcpDQo+IG1ha2VbMl06IEVudGVyaW5nIGRpcmVjdG9yeSBgL2hv
bWUvdXNlci9Nb2xwcm8vc3JjL21waWNoMi0xLjQvdGVzdC9tcGknDQo+IC4vcnVudGVzdHMgLXNy
Y2Rpcj0uIC10ZXN0cz10ZXN0bGlzdCBcDQo+ICAgICAgICAgICAgICAgICAgICAgLW1waWV4ZWM9
L2hvbWUvdXNlci9Nb2xwcm8vc3JjL21waWNoMi1pbnN0YWxsL2Jpbi9tcGlleGVjIFwNCj4gICAg
ICAgICAgICAgICAgICAgICAteG1sZmlsZT1zdW1tYXJ5LnhtbA0KPiBMb29raW5nIGluIC4vdGVz
dGxpc3QNCj4gUHJvY2Vzc2luZyBkaXJlY3RvcnkgYXR0cg0KPiBMb29raW5nIGluIC4vYXR0ci90
ZXN0bGlzdA0KPiBQcm9jZXNzaW5nIGRpcmVjdG9yeSBjb2xsDQo+IExvb2tpbmcgaW4gLi9jb2xs
L3Rlc3RsaXN0DQo+IFVuZXhwZWN0ZWQgb3V0cHV0IGluIGFsbHJlZDogW21waWV4ZWNAbm9kZTAx
XSBBUFBMSUNBVElPTiBUSU1FRCBPVVQNCj4gVW5leHBlY3RlZCBvdXRwdXQgaW4gYWxscmVkOiBb
cHJveHk6MDowQG5vZGUwMV0gSFlEX3BtY2RfcG1pcF9jb250cm9sX2NtZF9jYg0KPiAoLi9wbS9w
bWlzZXJ2L3BtaXBfY2IuYzo5MDYpOiBhc3NlcnQgKCFjbG9zZWQpIGZhaWxlZA0KPiBVbmV4cGVj
dGVkIG91dHB1dCBpbiBhbGxyZWQ6IFtwcm94eTowOjBAbm9kZTAxXSBIWURUX2RteHVfcG9sbF93
YWl0X2Zvcl9ldmVudA0KPiAoLi90b29scy9kZW11eC9kZW11eF9wb2xsLmM6NzcpOiBjYWxsYmFj
ayByZXR1cm5lZCBlcnJvciBzdGF0dXMNCj4gVW5leHBlY3RlZCBvdXRwdXQgaW4gYWxscmVkOiBb
cHJveHk6MDowQG5vZGUwMV0gbWFpbiANCj4gKC4vcG0vcG1pc2Vydi9wbWlwLmM6MjI2KToNCj4g
ZGVtdXggZW5naW5lIGVycm9yIHdhaXRpbmcgZm9yIGV2ZW50DQo+IFVuZXhwZWN0ZWQgb3V0cHV0
IGluIGFsbHJlZDogW21waWV4ZWNAbm9kZTAxXSBIWURUX2JzY3Vfd2FpdF9mb3JfY29tcGxldGlv
bg0KPiAoLi90b29scy9ib290c3RyYXAvdXRpbHMvYnNjdV93YWl0LmM6NzApOiBvbmUgb2YgdGhl
IHByb2Nlc3NlcyB0ZXJtaW5hdGVkDQo+IGJhZGx5OyBhYm9ydGluZw0KPiBVbmV4cGVjdGVkIG91
dHB1dCBpbiBhbGxyZWQ6IFttcGlleGVjQG5vZGUwMV0gSFlEVF9ic2NpX3dhaXRfZm9yX2NvbXBs
ZXRpb24NCj4gKC4vdG9vbHMvYm9vdHN0cmFwL3NyYy9ic2NpX3dhaXQuYzoyMyk6IGxhdW5jaGVy
IHJldHVybmVkIGVycm9yIHdhaXRpbmcgZm9yDQo+IGNvbXBsZXRpb24NCj4gVW5leHBlY3RlZCBv
dXRwdXQgaW4gYWxscmVkOiBbbXBpZXhlY0Bub2RlMDFdIEhZRF9wbWNpX3dhaXRfZm9yX2NvbXBs
ZXRpb24NCj4gKC4vcG0vcG1pc2Vydi9wbWlzZXJ2X3BtY2kuYzoxODkpOiBsYXVuY2hlciByZXR1
cm5lZCBlcnJvciB3YWl0aW5nIGZvcg0KPiBjb21wbGV0aW9uDQo+IFVuZXhwZWN0ZWQgb3V0cHV0
IGluIGFsbHJlZDogW21waWV4ZWNAbm9kZTAxXSBtYWluICguL3VpL21waWNoL21waWV4ZWMuYzoz
OTcpOg0KPiBwcm9jZXNzIG1hbmFnZXIgZXJyb3Igd2FpdGluZyBmb3IgY29tcGxldGlvbg0KPiBQ
cm9ncmFtIGFsbHJlZCBleGl0ZWQgd2l0aG91dCBObyBFcnJvcnMNCj4gDQo+Pg0KPj4gSGkgR3Jl
Z29yeQ0KPj4NCj4+IFdlIGFyZSB1c2luZyBNUElDSDItMS4zLjEgYW5kIHRoZSBJT1IgbXBpY2gg
dGVzdC4gYXMgd2VsbCBhcyB0aGUgbWR0ZXN0DQo+PiB0ZXN0LiBBbmQgaGF2ZSBoYWQgbm8gaXNz
dWVzIHNvIGZhciB3aXRoIG5mc3Y0IG5mc3Y0LjEgYW5kIHBuZnMuIEluIGZhY3QNCj4+IHRoaXMg
aXMgb3VyIHN0YW5kYXJkIHBlcmZvcm1hbmNlIHRlc3QuDQo+Pg0KPj4gV2hhdCB0ZXN0cyBhcmUg
eW91IHVzaW5nPw0KPj4gRG8geW91IGtub3cgb2YgYW55IG1ham9yIGNoYW5nZXMgYmV0d2VlbiBN
UElDSDItMS4zLjEgYW5kIE1QSUNIMi0xLjQ/DQo+PiBBbHNvIGFyZSB5b3UgdXNpbmcgdGhlIGJ1
aWx0aW4gbmZzLWNsaWVudCBkcml2ZXIgb3IgdGhlIFBPU0lYIGludGVyZmFjZT8NCj4+DQo+PiBC
b2F6DQo+Pg0KPiANCj4gDQoNCgTvv717Lm7vv70r77+977+977+977+977+977+977+9KyXvv73v
v71sendt77+977+9Yu+/veunsu+/ve+/vXLvv73vv716WO+/ve+/vRnfsinvv73vv73vv713Kh9q
Z++/ve+/ve+/vR7vv73vv73vv73vv73vv73domov77+977+977+9eu+/vd6W77+977+9Mu+/vd6Z
77+977+977+9Ju+/vSnfoe+/vWHvv73vv71/77+977+9Hu+/vUfvv73vv73vv71o77+9D++/vWo6
K3bvv73vv73vv71377+92aU=

RE: NFSv4 vs NFSv3 with MPICH2-1.4

[Gregory Magoon]

Thanks all for the feedback and sorry for the delay...one of our HDDs 
failed on
Saturday, so I had to take care of that.

Because I don't want to interrupt a working system, it will not be convenient
for me to try the "no delegations" option that has been suggested.

I was however, able to grab a hold of a temporarily free node (temporarily
returned to NFSv4 configuration) to capture the tcp traffic. I have sent a
short (< 1 sec) snapshot captured during (I believe) the allred3 mpich2 
test. I
have privately sent you a link to the file. Hopefully the issue will be 
obvious
from this (e.g. you will immediately see that I am doing something I shouldn't
be doing). If a longer snapshot started before the tests would be 
useful, I can
get that too.

I had posted on the mpich mailing list before I came here (
http://lists.mcs.anl.gov/pipermail/mpich-discuss/2011-July/010432.html ) and
unfortunately they weren't able to provide any insights.

Thanks again,
Greg


Quoting "Loewe, Bill" <bloewe@panasas.com>:

> Hi Greg,
>
> IOR is independent of MPICH2, but does require MPI for process 
> coordination.  By default, IOR will use the "-a POSIX" option for 
> standard POSIX I/O -- open(), write(), close(), etc.
>
> In addition, IOR can use the MPI-IO library calls (MPI_File_open(), 
> etc.) to perform I/O.
>
> For the build process of MPICH2 "make tests" exercises this MPI-IO 
> (ROMIO) interface which uses an ADIO (Abstract-Device Interface for 
> I/O) layer.  ADIO can interface to different file systems (NFS, 
> PanFS, PVFS2, Lustre, e.g.).
>
> The errors you're encountering in "make tests" for MPICH2 do not 
> appear to be testing the I/O, however, but seem to be an issue with 
> the launcher for the tests in general.  I agree with Boaz that it may 
> make sense to follow up with the MPICH developers for this.  Under 
> their main page (http://www.mcs.anl.gov/research/projects/mpich2/) 
> they have a support pulldown with FAQ and a mailing list.  They may 
> be able to help resolve this for you.
>
> Thanks,
>
> --Bill.
>
> -----Original Message-----
> From: Harrosh, Boaz
> Sent: Friday, July 29, 2011 8:20 PM
> To: Gregory Magoon
> Cc: Trond Myklebust; linux-nfs@vger.kernel.org; J. Bruce Fields; Loewe, Bill
> Subject: Re: NFSv4 vs NFSv3 with MPICH2-1.4
>
> On 07/28/2011 04:15 PM, Gregory Magoon wrote:
>> Unfortunately, I'm not familiar enough with MPICH2 to have an idea about
>> significant changes between version 1.3 and 1.4, but other evidence suggests
>> that the version is not the issue and that I would have the same 
>> problem with
>> v1.3.
>>
>> I'm using the MPICH2 test suite invoked by "make testing" (see below
>> for initial
>> output).
>>
>> I'm using the nfs-kernel-server and nfs-common Ubuntu packages (natty
>> release).
>>
>
> You have not answered the most important question:
>>> Also are you using the builtin nfs-client driver or the POSIX interface?
>
> Which I'll assume means you don't know. So I'll try to elaborate. Just for
> background, I've never used "make tests" before all I used was IOR & mdtest.
>
> Now if you print the usage string for IOR you get this option:
>
> 	-a S  api --  API for I/O [POSIX|MPIIO|HDF5|NCMPI]
>
> I'm not familiar with the code but what I understand is only "-a 
> POSIX" will actually
> use the regular Kernel VFS interface for read/writing of files. The 
> other options
> have different drivers for different protocols. I do not know first 
> hand, but I once
> heard in a conference that -a MPIIO has a special NFS driver that 
> uses better NFS
> semantics and avoids the POSIX semantics which are bad for big 
> cluster performance.
> All this is speculations and rumors on my part, and you will need to 
> consult with the
> mpich guys.
>
> Now I can imagine that a "make tests" would try all possible 
> combinations of "-a S"
> So you'll need to dig out what is the falling test and is it really 
> using the Kernel
> NFS driver at that point. (I bet if you do a tcpdump like Bruce said 
> the guys here will
> be able to see if this is a Linux NFS or not)
>
> I CC: Bill Loewe that might know much more then me about this 
> subject. And please do
> speak with the MPICH people (But keep us in the loop it is 
> interesting to know)
>
> Thanks
> Boaz
>
>> Thanks,
>> Greg
>>
>> user@node01:~/Molpro/src/mpich2-1.4$ make testing
>> (cd test && make testing)
>> make[1]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test'
>> (NOXMLCLOSE=YES && export NOXMLCLOSE && cd mpi && make testing)
>> make[2]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test/mpi'
>> ./runtests -srcdir=. -tests=testlist \
>>                     
>> -mpiexec=/home/user/Molpro/src/mpich2-install/bin/mpiexec \
>>                     -xmlfile=summary.xml
>> Looking in ./testlist
>> Processing directory attr
>> Looking in ./attr/testlist
>> Processing directory coll
>> Looking in ./coll/testlist
>> Unexpected output in allred: [mpiexec@node01] APPLICATION TIMED OUT
>> Unexpected output in allred: [proxy:0:0@node01] HYD_pmcd_pmip_control_cmd_cb
>> (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>> Unexpected output in allred: [proxy:0:0@node01] 
>> HYDT_dmxu_poll_wait_for_event
>> (./tools/demux/demux_poll.c:77): callback returned error status
>> Unexpected output in allred: [proxy:0:0@node01] main
>> (./pm/pmiserv/pmip.c:226):
>> demux engine error waiting for event
>> Unexpected output in allred: [mpiexec@node01] HYDT_bscu_wait_for_completion
>> (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated
>> badly; aborting
>> Unexpected output in allred: [mpiexec@node01] HYDT_bsci_wait_for_completion
>> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for
>> completion
>> Unexpected output in allred: [mpiexec@node01] HYD_pmci_wait_for_completion
>> (./pm/pmiserv/pmiserv_pmci.c:189): launcher returned error waiting for
>> completion
>> Unexpected output in allred: [mpiexec@node01] main 
>> (./ui/mpich/mpiexec.c:397):
>> process manager error waiting for completion
>> Program allred exited without No Errors
>>
>>>
>>> Hi Gregory
>>>
>>> We are using MPICH2-1.3.1 and the IOR mpich test. as well as the mdtest
>>> test. And have had no issues so far with nfsv4 nfsv4.1 and pnfs. In fact
>>> this is our standard performance test.
>>>
>>> What tests are you using?
>>> Do you know of any major changes between MPICH2-1.3.1 and MPICH2-1.4?
>>> Also are you using the builtin nfs-client driver or the POSIX interface?
>>>
>>> Boaz
>>>
>>
>>
>
>

RE: NFSv4 vs NFSv3 with MPICH2-1.4

[Gregory Magoon]

Just a quick follow-up...I was wondering if anyone had the chance to 
take a look
at the tcpdump I sent to a few of you last week.

If anyone else on the list wants to take a look, please let me know, 
and I will
send you the link privately.

Thanks,
Greg

Quoting Gregory Magoon <gmagoon@MIT.EDU>:

> Thanks all for the feedback and sorry for the delay...one of our HDDs 
> failed on
> Saturday, so I had to take care of that.
>
> Because I don't want to interrupt a working system, it will not be convenient
> for me to try the "no delegations" option that has been suggested.
>
> I was however, able to grab a hold of a temporarily free node (temporarily
> returned to NFSv4 configuration) to capture the tcp traffic. I have sent a
> short (< 1 sec) snapshot captured during (I believe) the allred3 
> mpich2 test. I
> have privately sent you a link to the file. Hopefully the issue will 
> be obvious
> from this (e.g. you will immediately see that I am doing something I 
> shouldn't
> be doing). If a longer snapshot started before the tests would be 
> useful, I can
> get that too.
>
> I had posted on the mpich mailing list before I came here (
> http://lists.mcs.anl.gov/pipermail/mpich-discuss/2011-July/010432.html ) and
> unfortunately they weren't able to provide any insights.
>
> Thanks again,
> Greg
>
>
> Quoting "Loewe, Bill" <bloewe@panasas.com>:
>
>> Hi Greg,
>>
>> IOR is independent of MPICH2, but does require MPI for process 
>> coordination.  By default, IOR will use the "-a POSIX" option for 
>> standard POSIX I/O -- open(), write(), close(), etc.
>>
>> In addition, IOR can use the MPI-IO library calls (MPI_File_open(), 
>> etc.) to perform I/O.
>>
>> For the build process of MPICH2 "make tests" exercises this MPI-IO 
>> (ROMIO) interface which uses an ADIO (Abstract-Device Interface for 
>> I/O) layer.  ADIO can interface to different file systems (NFS, 
>> PanFS, PVFS2, Lustre, e.g.).
>>
>> The errors you're encountering in "make tests" for MPICH2 do not 
>> appear to be testing the I/O, however, but seem to be an issue with 
>> the launcher for the tests in general.  I agree with Boaz that it 
>> may make sense to follow up with the MPICH developers for this.  
>> Under their main page 
>> (http://www.mcs.anl.gov/research/projects/mpich2/) they have a 
>> support pulldown with FAQ and a mailing list.  They may be able to 
>> help resolve this for you.
>>
>> Thanks,
>>
>> --Bill.
>>
>> -----Original Message-----
>> From: Harrosh, Boaz
>> Sent: Friday, July 29, 2011 8:20 PM
>> To: Gregory Magoon
>> Cc: Trond Myklebust; linux-nfs@vger.kernel.org; J. Bruce Fields; Loewe, Bill
>> Subject: Re: NFSv4 vs NFSv3 with MPICH2-1.4
>>
>> On 07/28/2011 04:15 PM, Gregory Magoon wrote:
>>> Unfortunately, I'm not familiar enough with MPICH2 to have an idea about
>>> significant changes between version 1.3 and 1.4, but other evidence 
>>> suggests
>>> that the version is not the issue and that I would have the same 
>>> problem with
>>> v1.3.
>>>
>>> I'm using the MPICH2 test suite invoked by "make testing" (see below
>>> for initial
>>> output).
>>>
>>> I'm using the nfs-kernel-server and nfs-common Ubuntu packages (natty
>>> release).
>>>
>>
>> You have not answered the most important question:
>>>> Also are you using the builtin nfs-client driver or the POSIX interface?
>>
>> Which I'll assume means you don't know. So I'll try to elaborate. Just for
>> background, I've never used "make tests" before all I used was IOR & mdtest.
>>
>> Now if you print the usage string for IOR you get this option:
>>
>> 	-a S  api --  API for I/O [POSIX|MPIIO|HDF5|NCMPI]
>>
>> I'm not familiar with the code but what I understand is only "-a 
>> POSIX" will actually
>> use the regular Kernel VFS interface for read/writing of files. The 
>> other options
>> have different drivers for different protocols. I do not know first 
>> hand, but I once
>> heard in a conference that -a MPIIO has a special NFS driver that 
>> uses better NFS
>> semantics and avoids the POSIX semantics which are bad for big 
>> cluster performance.
>> All this is speculations and rumors on my part, and you will need to 
>> consult with the
>> mpich guys.
>>
>> Now I can imagine that a "make tests" would try all possible 
>> combinations of "-a S"
>> So you'll need to dig out what is the falling test and is it really 
>> using the Kernel
>> NFS driver at that point. (I bet if you do a tcpdump like Bruce said 
>> the guys here will
>> be able to see if this is a Linux NFS or not)
>>
>> I CC: Bill Loewe that might know much more then me about this 
>> subject. And please do
>> speak with the MPICH people (But keep us in the loop it is 
>> interesting to know)
>>
>> Thanks
>> Boaz
>>
>>> Thanks,
>>> Greg
>>>
>>> user@node01:~/Molpro/src/mpich2-1.4$ make testing
>>> (cd test && make testing)
>>> make[1]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test'
>>> (NOXMLCLOSE=YES && export NOXMLCLOSE && cd mpi && make testing)
>>> make[2]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test/mpi'
>>> ./runtests -srcdir=. -tests=testlist \
>>>                     
>>> -mpiexec=/home/user/Molpro/src/mpich2-install/bin/mpiexec \
>>>                     -xmlfile=summary.xml
>>> Looking in ./testlist
>>> Processing directory attr
>>> Looking in ./attr/testlist
>>> Processing directory coll
>>> Looking in ./coll/testlist
>>> Unexpected output in allred: [mpiexec@node01] APPLICATION TIMED OUT
>>> Unexpected output in allred: [proxy:0:0@node01] 
>>> HYD_pmcd_pmip_control_cmd_cb
>>> (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>>> Unexpected output in allred: [proxy:0:0@node01] 
>>> HYDT_dmxu_poll_wait_for_event
>>> (./tools/demux/demux_poll.c:77): callback returned error status
>>> Unexpected output in allred: [proxy:0:0@node01] main
>>> (./pm/pmiserv/pmip.c:226):
>>> demux engine error waiting for event
>>> Unexpected output in allred: [mpiexec@node01] HYDT_bscu_wait_for_completion
>>> (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated
>>> badly; aborting
>>> Unexpected output in allred: [mpiexec@node01] HYDT_bsci_wait_for_completion
>>> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for
>>> completion
>>> Unexpected output in allred: [mpiexec@node01] HYD_pmci_wait_for_completion
>>> (./pm/pmiserv/pmiserv_pmci.c:189): launcher returned error waiting for
>>> completion
>>> Unexpected output in allred: [mpiexec@node01] main 
>>> (./ui/mpich/mpiexec.c:397):
>>> process manager error waiting for completion
>>> Program allred exited without No Errors
>>>
>>>>
>>>> Hi Gregory
>>>>
>>>> We are using MPICH2-1.3.1 and the IOR mpich test. as well as the mdtest
>>>> test. And have had no issues so far with nfsv4 nfsv4.1 and pnfs. In fact
>>>> this is our standard performance test.
>>>>
>>>> What tests are you using?
>>>> Do you know of any major changes between MPICH2-1.3.1 and MPICH2-1.4?
>>>> Also are you using the builtin nfs-client driver or the POSIX interface?
>>>>
>>>> Boaz
>>>>
>>>
>>>
>>
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: NFSv4 vs NFSv3 with MPICH2-1.4

[Boaz Harrosh]

On 08/09/2011 04:15 PM, Gregory Magoon wrote:
> Just a quick follow-up...I was wondering if anyone had the chance to 
> take a look
> at the tcpdump I sent to a few of you last week.
> 
> If anyone else on the list wants to take a look, please let me know, 
> and I will
> send you the link privately.
> 

I will not have any bandwidth to look at the logs any time soon.

One thing that you should do is find out what is the exact test that is
failing. And is it really using the Linux-Kernel nfs client or if it is
using mpi's private nfs client.

Please find out the exact mpi command that is ran and all its command line
switches it uses.

Try to reproduce the problem not with "make test" but with the exact command
above that fails. Send me the command I'll have a look.

Boaz

> Thanks,
> Greg
> 
> Quoting Gregory Magoon <gmagoon@MIT.EDU>:
> 
>> Thanks all for the feedback and sorry for the delay...one of our HDDs 
>> failed on
>> Saturday, so I had to take care of that.
>>
>> Because I don't want to interrupt a working system, it will not be convenient
>> for me to try the "no delegations" option that has been suggested.
>>
>> I was however, able to grab a hold of a temporarily free node (temporarily
>> returned to NFSv4 configuration) to capture the tcp traffic. I have sent a
>> short (< 1 sec) snapshot captured during (I believe) the allred3 
>> mpich2 test. I
>> have privately sent you a link to the file. Hopefully the issue will 
>> be obvious
>> from this (e.g. you will immediately see that I am doing something I 
>> shouldn't
>> be doing). If a longer snapshot started before the tests would be 
>> useful, I can
>> get that too.
>>
>> I had posted on the mpich mailing list before I came here (
>> http://lists.mcs.anl.gov/pipermail/mpich-discuss/2011-July/010432.html ) and
>> unfortunately they weren't able to provide any insights.
>>
>> Thanks again,
>> Greg
>>
>>
>> Quoting "Loewe, Bill" <bloewe@panasas.com>:
>>
>>> Hi Greg,
>>>
>>> IOR is independent of MPICH2, but does require MPI for process 
>>> coordination.  By default, IOR will use the "-a POSIX" option for 
>>> standard POSIX I/O -- open(), write(), close(), etc.
>>>
>>> In addition, IOR can use the MPI-IO library calls (MPI_File_open(), 
>>> etc.) to perform I/O.
>>>
>>> For the build process of MPICH2 "make tests" exercises this MPI-IO 
>>> (ROMIO) interface which uses an ADIO (Abstract-Device Interface for 
>>> I/O) layer.  ADIO can interface to different file systems (NFS, 
>>> PanFS, PVFS2, Lustre, e.g.).
>>>
>>> The errors you're encountering in "make tests" for MPICH2 do not 
>>> appear to be testing the I/O, however, but seem to be an issue with 
>>> the launcher for the tests in general.  I agree with Boaz that it 
>>> may make sense to follow up with the MPICH developers for this.  
>>> Under their main page 
>>> (http://www.mcs.anl.gov/research/projects/mpich2/) they have a 
>>> support pulldown with FAQ and a mailing list.  They may be able to 
>>> help resolve this for you.
>>>
>>> Thanks,
>>>
>>> --Bill.
>>>
>>> -----Original Message-----
>>> From: Harrosh, Boaz
>>> Sent: Friday, July 29, 2011 8:20 PM
>>> To: Gregory Magoon
>>> Cc: Trond Myklebust; linux-nfs@vger.kernel.org; J. Bruce Fields; Loewe, Bill
>>> Subject: Re: NFSv4 vs NFSv3 with MPICH2-1.4
>>>
>>> On 07/28/2011 04:15 PM, Gregory Magoon wrote:
>>>> Unfortunately, I'm not familiar enough with MPICH2 to have an idea about
>>>> significant changes between version 1.3 and 1.4, but other evidence 
>>>> suggests
>>>> that the version is not the issue and that I would have the same 
>>>> problem with
>>>> v1.3.
>>>>
>>>> I'm using the MPICH2 test suite invoked by "make testing" (see below
>>>> for initial
>>>> output).
>>>>
>>>> I'm using the nfs-kernel-server and nfs-common Ubuntu packages (natty
>>>> release).
>>>>
>>>
>>> You have not answered the most important question:
>>>>> Also are you using the builtin nfs-client driver or the POSIX interface?
>>>
>>> Which I'll assume means you don't know. So I'll try to elaborate. Just for
>>> background, I've never used "make tests" before all I used was IOR & mdtest.
>>>
>>> Now if you print the usage string for IOR you get this option:
>>>
>>> 	-a S  api --  API for I/O [POSIX|MPIIO|HDF5|NCMPI]
>>>
>>> I'm not familiar with the code but what I understand is only "-a 
>>> POSIX" will actually
>>> use the regular Kernel VFS interface for read/writing of files. The 
>>> other options
>>> have different drivers for different protocols. I do not know first 
>>> hand, but I once
>>> heard in a conference that -a MPIIO has a special NFS driver that 
>>> uses better NFS
>>> semantics and avoids the POSIX semantics which are bad for big 
>>> cluster performance.
>>> All this is speculations and rumors on my part, and you will need to 
>>> consult with the
>>> mpich guys.
>>>
>>> Now I can imagine that a "make tests" would try all possible 
>>> combinations of "-a S"
>>> So you'll need to dig out what is the falling test and is it really 
>>> using the Kernel
>>> NFS driver at that point. (I bet if you do a tcpdump like Bruce said 
>>> the guys here will
>>> be able to see if this is a Linux NFS or not)
>>>
>>> I CC: Bill Loewe that might know much more then me about this 
>>> subject. And please do
>>> speak with the MPICH people (But keep us in the loop it is 
>>> interesting to know)
>>>
>>> Thanks
>>> Boaz
>>>
>>>> Thanks,
>>>> Greg
>>>>
>>>> user@node01:~/Molpro/src/mpich2-1.4$ make testing
>>>> (cd test && make testing)
>>>> make[1]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test'
>>>> (NOXMLCLOSE=YES && export NOXMLCLOSE && cd mpi && make testing)
>>>> make[2]: Entering directory `/home/user/Molpro/src/mpich2-1.4/test/mpi'
>>>> ./runtests -srcdir=. -tests=testlist \
>>>>                     
>>>> -mpiexec=/home/user/Molpro/src/mpich2-install/bin/mpiexec \
>>>>                     -xmlfile=summary.xml
>>>> Looking in ./testlist
>>>> Processing directory attr
>>>> Looking in ./attr/testlist
>>>> Processing directory coll
>>>> Looking in ./coll/testlist
>>>> Unexpected output in allred: [mpiexec@node01] APPLICATION TIMED OUT
>>>> Unexpected output in allred: [proxy:0:0@node01] 
>>>> HYD_pmcd_pmip_control_cmd_cb
>>>> (./pm/pmiserv/pmip_cb.c:906): assert (!closed) failed
>>>> Unexpected output in allred: [proxy:0:0@node01] 
>>>> HYDT_dmxu_poll_wait_for_event
>>>> (./tools/demux/demux_poll.c:77): callback returned error status
>>>> Unexpected output in allred: [proxy:0:0@node01] main
>>>> (./pm/pmiserv/pmip.c:226):
>>>> demux engine error waiting for event
>>>> Unexpected output in allred: [mpiexec@node01] HYDT_bscu_wait_for_completion
>>>> (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated
>>>> badly; aborting
>>>> Unexpected output in allred: [mpiexec@node01] HYDT_bsci_wait_for_completion
>>>> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for
>>>> completion
>>>> Unexpected output in allred: [mpiexec@node01] HYD_pmci_wait_for_completion
>>>> (./pm/pmiserv/pmiserv_pmci.c:189): launcher returned error waiting for
>>>> completion
>>>> Unexpected output in allred: [mpiexec@node01] main 
>>>> (./ui/mpich/mpiexec.c:397):
>>>> process manager error waiting for completion
>>>> Program allred exited without No Errors
>>>>
>>>>>
>>>>> Hi Gregory
>>>>>
>>>>> We are using MPICH2-1.3.1 and the IOR mpich test. as well as the mdtest
>>>>> test. And have had no issues so far with nfsv4 nfsv4.1 and pnfs. In fact
>>>>> this is our standard performance test.
>>>>>
>>>>> What tests are you using?
>>>>> Do you know of any major changes between MPICH2-1.3.1 and MPICH2-1.4?
>>>>> Also are you using the builtin nfs-client driver or the POSIX interface?
>>>>>
>>>>> Boaz
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> 

Re: NFSv4 vs NFSv3 with MPICH2-1.4

[J. Bruce Fields]

On Thu, Jul 28, 2011 at 05:47:57PM -0400, Trond Myklebust wrote:
> On Thu, 2011-07-28 at 17:24 -0400, Gregory Magoon wrote: 
> > Thanks for the tips...unfortunately, making the changes you suggest (removing
> > timeo, rsize, wsize options) doesn't seem to address the issue with MPICH2 and
> > NFSv4.
> 
> Have you turned off delegations on the server? I wouldn't expect them to
> help much on an MPI workload.

Note, you can do that with "echo 0 >/proc/sys/fs/leases-enable before
starting nfsd.

> Otherwise, you might want to post a comparison of your results from
> 'nfsstat' for your workload on NFSv3 and NFSv4.

Yes.

Taking a sample of the network traffic once it gets stuck might also be
interesting.

(Wait for it to get stuck, the run "tcpdump -s0 -wtmp.pcap", let it go
for a second (longer if that doesn't get anything), then interrupt it
and send us tmp.pcap.)

Or if it gets stuck immediately, then you could start the tcpdump before
the you start your tests and capture everything.  But if it doesn't get
stuck immediately that could be a ton of data.

--b.