limit on filesystem size?

limit on filesystem size?

[Martin Siegert]

Hello,

we are in the planning process for the setup of a large (~ 100 TB) NFS
server. My understanding is that there is a limit of 2TB for filesystem
sizes for local (!) filesystems under Linux (at least for ext2/ext3;
I am not sure about xfs).

My question really is: is there a limit for the filesystem size that
a Linux NFS client (2.4.18 or the upcoming 2.4.19 kernel) can mount
from a (possibly non-Linux) NFS server?
E.g., can I mount a 20TB filesystem under Linux?

(I've asked this question on the beowulf mailing list and the answer was
no. Now I ask the experts - hoping for a different answer :-)

Thanks a lot for your help!

Cheers,
Martin

========================================================================
Martin Siegert
Academic Computing Services                        phone: (604) 291-4691
Simon Fraser University                            fax:   (604) 291-4242
Burnaby, British Columbia                          email: siegert@sfu.ca
Canada  V5A 1S6
========================================================================


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: limit on filesystem size?

[Martin Siegert]

Thanks, Trond.

On Thu, Aug 01, 2002 at 10:39:53PM +0200, Trond Myklebust wrote:
> >>>>> " " == Martin Siegert <siegert@sfu.ca> writes:
> 
>      > Hello, we are in the planning process for the setup of a large
>      > (~ 100 TB) NFS server. My understanding is that there is a
>      > limit of 2TB for filesystem sizes for local (!) filesystems
>      > under Linux (at least for ext2/ext3; I am not sure about xfs).
> 
>      > My question really is: is there a limit for the filesystem size
>      > that a Linux NFS client (2.4.18 or the upcoming 2.4.19 kernel)
>      > can mount from a (possibly non-Linux) NFS server?  E.g., can I
>      > mount a 20TB filesystem under Linux?
> 
>      > (I've asked this question on the beowulf mailing list and the
>      > answer was no. Now I ask the experts - hoping for a different
>      > answer :-)
> 
> If you can mount it, then you can export it...

Please, bear with me - I am by no means an expert on these issues.
But my impression was that the problem is in include/linux/blkdev.h

struct request {
...
        unsigned long sector;

with results in a (local) filesystem size limit of 2TB (unless sectors
larger than 512 bytes are used).

Am I understanding correctly that you are telling me that this does not
apply to NFS and that despite this definition I could even reexport
(user space NFS?) such a filesystem?

> For NFSv2 there is a 32-bit size limit on the files you can export
> whereas for NFSv3, that limit is 64-bits, but those limits are
> *per-file*.

Yes, I do not have any problems anymore with files larger than 2GB.

> Otherwise, you might run into problems when you try to run 'df' in
> order to read filesystem statistics (NFSv2, the limit is 41-bit
> - NFSv3 the limit is 64-bit) but that shouldn't really be a critical
>  feature.

Agreed. That wouldn't be a big problem. As long as I can read and write
to such a 20TB NFS mounted filesystem everything would be secondary.

Thanks again,

Cheers,
Martin


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: limit on filesystem size?

[Trond Myklebust]

>>>>> " " == Martin Siegert <siegert@sfu.ca> writes:

     > Please, bear with me - I am by no means an expert on these
     > issues.  But my impression was that the problem is in
     > include/linux/blkdev.h

     > struct request { ...
     >         unsigned long sector;

     > with results in a (local) filesystem size limit of 2TB (unless
     > sectors larger than 512 bytes are used).

     > Am I understanding correctly that you are telling me that this
     > does not apply to NFS and that despite this definition I could
     > even reexport (user space NFS?) such a filesystem?

I repeat: yes!

NFS does not know anything about sectors. It indexes *files* by means
of an opaque 32-byte (NFSv2) or 64-byte (v3) filehandle. The latter
can be considered to be the equivalent of 'file descriptors' as used
by libc does in open()/read()/write().

Rexporting filesystems is never a good idea though. Better to allow
all NFS clients direct access to the server...

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: limit on filesystem size?

[Steve Costaras]

How are you planning on creating this filesystem?    I have tried
creating 2TB filessystems here for a while and have had little or no
luck w/ Solaris or linux.  Using Disksuite for solaris and LVM and MD for
linux no luck.  I have not tried veritas VM yet but even calling up SUN
the answer is that 1TB is the limit per filesystem currently under solaris 8
& 9

Linux is really around 980GB or so where it starts to flake out.


Steve
----- Original Message -----
From: "Trond Myklebust" <trond.myklebust@fys.uio.no>
To: "Martin Siegert" <siegert@sfu.ca>
Cc: <nfs@lists.sourceforge.net>
Sent: Thursday, August 01, 2002 15:39
Subject: Re: [NFS] limit on filesystem size?


> >>>>> " " == Martin Siegert <siegert@sfu.ca> writes:
>
>      > Hello, we are in the planning process for the setup of a large
>      > (~ 100 TB) NFS server. My understanding is that there is a
>      > limit of 2TB for filesystem sizes for local (!) filesystems
>      > under Linux (at least for ext2/ext3; I am not sure about xfs).
>
>      > My question really is: is there a limit for the filesystem size
>      > that a Linux NFS client (2.4.18 or the upcoming 2.4.19 kernel)
>      > can mount from a (possibly non-Linux) NFS server?  E.g., can I
>      > mount a 20TB filesystem under Linux?
>
>      > (I've asked this question on the beowulf mailing list and the
>      > answer was no. Now I ask the experts - hoping for a different
>      > answer :-)
>
> If you can mount it, then you can export it...
>
> For NFSv2 there is a 32-bit size limit on the files you can export
> whereas for NFSv3, that limit is 64-bits, but those limits are
> *per-file*.
>
> Otherwise, you might run into problems when you try to run 'df' in
> order to read filesystem statistics (NFSv2, the limit is 41-bit
> - NFSv3 the limit is 64-bit) but that shouldn't really be a critical
>  feature.
>
> Cheers,
>   Trond
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: limit on filesystem size?

[Ragnar Kjørstad]

On Fri, Aug 02, 2002 at 04:26:35PM -0500, Steve Costaras wrote:
> How are you planning on creating this filesystem?    I have tried
> creating 2TB filessystems here for a while and have had little or no
> luck w/ Solaris or linux.  Using Disksuite for solaris and LVM and MD f=
or
> linux no luck.  I have not tried veritas VM yet but even calling up SUN
> the answer is that 1TB is the limit per filesystem currently under sola=
ris 8
> & 9
>=20
> Linux is really around 980GB or so where it starts to flake out.

Linux goes up to 2 TB if you use a scsi-driver that use unsigned
sector-numbers and a recent version of LVM (or no LVM).



--=20
Ragnar Kj=F8rstad
Big Storage


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: limit on filesystem size?

[Martin Siegert]

On Fri, Aug 02, 2002 at 04:26:35PM -0500, Steve Costaras wrote:
> How are you planning on creating this filesystem?    I have tried
> creating 2TB filessystems here for a while and have had little or no
> luck w/ Solaris or linux.  Using Disksuite for solaris and LVM and MD for
> linux no luck.  I have not tried veritas VM yet but even calling up SUN
> the answer is that 1TB is the limit per filesystem currently under solaris 8
> & 9

My understanding is that you can create filesystems larger than 2TB under
Solaris 8. You almost certainly can do it under Irix.
But this is really what my question is about: we are shortly going to
write a RFP and I was wondering whether we should make it a requirement
that we can create filesystems on the NFS server that are larger than 2 TB.
If Linux clients wouldn't be able to mount such filesystems then this would
be irrelevant (because we must configure the NFS server such that Linux 
clients can mount the filesystems in any case). However, if Linux clients
can mount such filesystems, creating e.g., 4 20TB filesystems is definitely
easier to manage than 40 2TB filesystems. Thus in that case we would buy
a NFS server/storage solution that allows us to create such filesystems.

> Linux is really around 980GB or so where it starts to flake out.

... that's why the NFS server that we are going to buy is probably not
going to run Linux :-(

Martin

> 
> Steve
> ----- Original Message -----
> From: "Trond Myklebust" <trond.myklebust@fys.uio.no>
> To: "Martin Siegert" <siegert@sfu.ca>
> Cc: <nfs@lists.sourceforge.net>
> Sent: Thursday, August 01, 2002 15:39
> Subject: Re: [NFS] limit on filesystem size?
> 
> 
> > >>>>> " " == Martin Siegert <siegert@sfu.ca> writes:
> >
> >      > Hello, we are in the planning process for the setup of a large
> >      > (~ 100 TB) NFS server. My understanding is that there is a
> >      > limit of 2TB for filesystem sizes for local (!) filesystems
> >      > under Linux (at least for ext2/ext3; I am not sure about xfs).
> >
> >      > My question really is: is there a limit for the filesystem size
> >      > that a Linux NFS client (2.4.18 or the upcoming 2.4.19 kernel)
> >      > can mount from a (possibly non-Linux) NFS server?  E.g., can I
> >      > mount a 20TB filesystem under Linux?
> >
> >      > (I've asked this question on the beowulf mailing list and the
> >      > answer was no. Now I ask the experts - hoping for a different
> >      > answer :-)
> >
> > If you can mount it, then you can export it...
> >
> > For NFSv2 there is a 32-bit size limit on the files you can export
> > whereas for NFSv3, that limit is 64-bits, but those limits are
> > *per-file*.
> >
> > Otherwise, you might run into problems when you try to run 'df' in
> > order to read filesystem statistics (NFSv2, the limit is 41-bit
> > - NFSv3 the limit is 64-bit) but that shouldn't really be a critical
> >  feature.
> >
> > Cheers,
> >   Trond
> >
> >
> > -------------------------------------------------------
> > This sf.net email is sponsored by:ThinkGeek
> > Welcome to geek heaven.
> > http://thinkgeek.com/sf
> > _______________________________________________
> > NFS maillist  -  NFS@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfs


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs