issue with mounting ceph cluster over NFS

issue with mounting ceph cluster over NFS

[madhusudhana U]

Hi,
I have a ceph cluster with 5 nodes, in which 2 are MDS, 3 are MON and all 
5 acts as OSD. I have mounted the ceph cluster in one node in the cluster
and exported the mounted dir via NFS. Below is my mount and exports file 
looks like


ceph-fuse on /ceph_cluster type fuse.ceph-fuse 
(rw,nosuid,nodev,allow_other,default_permissions)
[root@ceph-node-15 ~]# cat /etc/exports
/ceph_cluster   *(rw,no_root_squash,fsid=10001)

Below is automount entry

madhusudhan_ceph -
rw,intr,retrans=10,timeo=600,hard,rsize=32768,wsize=32768,tcp,noacl      ceph-
node-15:/ceph_cluster/madhusudhana_ceph

I am facing strange issue with one of my t_make build, where its failing
for some unknown reason. But the same build works fine on local machine and
build gets completed. There are no difference in the data as its been synced 
from perforce to both the directories.

Can someone put some lime light on the best way to mount the ceph cluster 
via NFS (using autofs to mount the dir) ? and is there anything that i need to 
make sure for mounting ceph cluster via NFS ? I heard that, t_make will fail
if underlying file system can't handle 64b file handles [inode number/fileid] 
(faced the same issue with isilon storage). Can ceph handle above condition ?

Any help/input greatly appreciated.

Thanks

Re: issue with mounting ceph cluster over NFS

[Josh Durgin]

On 05/15/2012 08:31 AM, madhusudhana U wrote:
> Hi,
> I have a ceph cluster with 5 nodes, in which 2 are MDS, 3 are MON and all

Is only one MDS active?

> 5 acts as OSD. I have mounted the ceph cluster in one node in the cluster
> and exported the mounted dir via NFS. Below is my mount and exports file
> looks like
>
>
> ceph-fuse on /ceph_cluster type fuse.ceph-fuse
> (rw,nosuid,nodev,allow_other,default_permissions)
> [root@ceph-node-15 ~]# cat /etc/exports
> /ceph_cluster   *(rw,no_root_squash,fsid=10001)
>
> Below is automount entry
>
> madhusudhan_ceph -
> rw,intr,retrans=10,timeo=600,hard,rsize=32768,wsize=32768,tcp,noacl      ceph-
> node-15:/ceph_cluster/madhusudhana_ceph
>
> I am facing strange issue with one of my t_make build, where its failing
> for some unknown reason. But the same build works fine on local machine and
> build gets completed. There are no difference in the data as its been synced
> from perforce to both the directories.

Does the build work on ceph-fuse without nfs?

> Can someone put some lime light on the best way to mount the ceph cluster
> via NFS (using autofs to mount the dir) ? and is there anything that i need to
> make sure for mounting ceph cluster via NFS ? I heard that, t_make will fail
> if underlying file system can't handle 64b file handles [inode number/fileid]
> (faced the same issue with isilon storage). Can ceph handle above condition ?

Ceph inodes are always 64-bit. The only exception is ceph-fuse on a
32-bit machine, which won't work in general due to the restricted inode
size.

Re: issue with mounting ceph cluster over NFS

[madhusudhana U]

> Is only one MDS active?
Yes only one MDS is active

> Does the build work on ceph-fuse without nfs?
I can't run build without NFS on ceph-fuse because, the build runs on a
cluster (we use LSF for it) where each machine mounts the directory. The 
build is spit into many small jobs [around 800 in number] and dispatched 
to LSF. LSF master host allocates different machines for a bunch of small
jobs. So, in order to successfully complete the build, each LSF host needs
to mount the ceph cluster in the same name( i.e, same mount point name, like
/home/madhusudhana_ceph). So, to get this work, I NFS exported ceph cluster 
from ceph-node-15 using autofs. Each node in LSF cluster has auto mounter map 
replicated used NIS.

> Ceph inodes are always 64-bit. The only exception is ceph-fuse on a
> 32-bit machine, which won't work in general due to the restricted inode
> size.
All my machines are 64b with centos 6.2 running on them. 

Re: issue with mounting ceph cluster over NFS

[Madhusudhana U]

madhusudhana U <madhusudhana.u.acharya <at> gmail.com> writes:

Any one had any clue on the issue which I am seeing ?