[linux-lvm] LVM limits?

[linux-lvm] LVM limits?

[Jordi Prats]

Hi all,
I've been browsing LVM's wikis but I found no reference to LVM's limits.

How large can I create a VG? And a LV?

Thanks,
Jordi

Re: [linux-lvm] LVM limits?

[Chris Cox]

On Mon, 2008-01-28 at 20:24 +0100, Jordi Prats wrote:
> Hi all,
> I found on a webpage this two sentences:
> 
> - The combination of 32-bit CPU and Linux kernel version 2.6.x, the
> limit of logical volume size is maximized at 16TB.
> 
> - For Linux kernel 2.6.x running on 64-bit CPU, the maximum LV size is
> 8EB (extremely terrible big storage for this time being!)
> 
> Are they right?

Ok.  But it's really impractical to have large multi-terabyte
single filesystem today.  What are you wanting to do?  Ever fsck a
2TB filesystem?  Consider yourself warned.

Or... were you just curious about the maximums?

Re: [linux-lvm] LVM limits?

[Ehud Karni]

On Mon, 28 Jan 2008 11:38:14 Chris Cox wrote:
>
> Ok.  But it's really impractical to have large multi-terabyte
> single filesystem today.  What are you wanting to do?  Ever fsck a
> 2TB filesystem?  Consider yourself warned.

Just last night I ran fsck on my home 1.5 TB file server (it is software
RAID-5 - mdadm, built on 7200 RPM, 500GB SATA2 x 4, ext3 without LVM).
It has went 191 days without fsck so when I booted the machine (I just
upgraded to 2.6.18-53.1.6.el5) it did the fsck automaticaly.

It took less then 90 minutes.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: [linux-lvm] LVM limits?

[Chris Cox]

On Mon, 2008-01-28 at 20:01 +0200, Ehud Karni wrote:
> On Mon, 28 Jan 2008 11:38:14 Chris Cox wrote:
> >
> > Ok.  But it's really impractical to have large multi-terabyte
> > single filesystem today.  What are you wanting to do?  Ever fsck a
> > 2TB filesystem?  Consider yourself warned.
> 
> Just last night I ran fsck on my home 1.5 TB file server (it is software
> RAID-5 - mdadm, built on 7200 RPM, 500GB SATA2 x 4, ext3 without LVM).
> It has went 191 days without fsck so when I booted the machine (I just
> upgraded to 2.6.18-53.1.6.el5) it did the fsck automaticaly.
> 
> It took less then 90 minutes.

:)  It can vary.  I've seen it take days (on infrastructure much
better than what you have).

Re: [linux-lvm] LVM limits?

[Jordi Prats]

Hi all,
I found on a webpage this two sentences:

- The combination of 32-bit CPU and Linux kernel version 2.6.x, the
limit of logical volume size is maximized at 16TB.

- For Linux kernel 2.6.x running on 64-bit CPU, the maximum LV size is
8EB (extremely terrible big storage for this time being!)

Are they right?

Thanks,
Jordi


Jordi Prats wrote:
> Hi all,
> I've been browsing LVM's wikis but I found no reference to LVM's limits.
> 
> How large can I create a VG? And a LV?
> 
> Thanks,
> Jordi
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 

Re: [linux-lvm] LVM limits?

[Vesa-Pekka Palmu]

On Mon, 28 Jan 2008, Ehud Karni wrote:

> On Mon, 28 Jan 2008 11:38:14 Chris Cox wrote:
>>
>> Ok.  But it's really impractical to have large multi-terabyte
>> single filesystem today.  What are you wanting to do?  Ever fsck a
>> 2TB filesystem?  Consider yourself warned.
>
> Just last night I ran fsck on my home 1.5 TB file server (it is software
> RAID-5 - mdadm, built on 7200 RPM, 500GB SATA2 x 4, ext3 without LVM).
> It has went 191 days without fsck so when I booted the machine (I just
> upgraded to 2.6.18-53.1.6.el5) it did the fsck automaticaly.
>
> It took less then 90 minutes.
>

ext3 isn't the best filesystem for large volumes, it's worth considering 
alternatives like jfs.

RE: [linux-lvm] LVM limits?

[Joseph L. Casale]

>It took less then 90 minutes.
>
>Ehud.

90 Minutes, how "enterprisable" is that? Just cause you can, doesn't mean you should: What if that server tanked during working hours at the corp? You would have to sit and wait for it to come back up. Worse than that are the rebuild times for degraded arrays. Try to rebuild a **huge** array on **huge** discs during production with lots of IO and hope you don't lose another disc. There are practical alternatives with thoughtful management to circumvent the need for and 8EB or larger array, but YMMV :)

jlc

Re: [linux-lvm] LVM limits?

[Ehud Karni]

On Mon, 28 Jan 2008 14:52:20 Joseph L. Casale wrote:
>
> >It took less then 90 minutes.
>
> 90 Minutes, how "enterprisable" is that? Just cause you can, doesn't
> mean you should: What if that server tanked during working hours at
> the corp? You would have to sit and wait for it to come back up.
> Worse than that are the rebuild times for degraded arrays. Try to
> rebuild a **huge** array on **huge** discs during production with
> lots of IO and hope you don't lose another disc. There are practical
> alternatives with thoughtful management to circumvent the need for
> and 8EB or larger array, but YMMV :)

First, that's on my home machine (the whole machine, including the
disks and controller, cost me about $1100).

At work I have 1.75 TB VG (RAID-5 using 3ware, 320 GB SATA-2 x 7, with
1 spare), but it is split into 3 LVs the largest of them is 700 GB.
I had several rebuilds, It takes about 14 hours with no disturbance
at all to the work.

The fsck problem can be dealt in various ways. Changing the FS may be
one of them but I have no experience in this way. Another, more ready
way is to tunefs the FSs (all ext3 in my case) so it will never occurs
automatically, and do scheduled fsck once or twice a year.

I think that when we'll have much larger arrays, their speed will also
be larger. For those who interested in ancient history, I recall that
12-13 years ago, I had a Clariion at work with just 40 GB (it was 10
scsi disks of 4GB) and the fsck (on Aviion DGIX) took more then 50
minutes. Morale - the advancement of technology work for us.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: [linux-lvm] LVM limits?

[Jordi Prats]

Hi,
I'm the system administrator of PADICAT (http://www.padi.cat). It
collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to
provide permanent access to them (http://www.padi.cat/en/quees.php).
It's equivalent to Internet Archive (http://www.archive.org) but for a
particular culture.

Our software developers require us to have one large file system,
actually a single directory, with all this historically-classified web
sites on a gziped file.

I'm currently studying lustre and other HPC-related file systems to get
this large file system, but by now I have ext3 as our file system. Next
Monday I'm planning to extend it to 3TB o 4TB, so I'm currently
researching for restrictions because during next month I'll have between
3TB to 4TB more to add: so, it will become a 8TB file system.

Last time I fsck my 2'1TB file system I spend about 2 hours. Anyway, I'm
also curious about the maximums :P

Thanks,
Jordi


Chris Cox wrote:
> On Mon, 2008-01-28 at 20:24 +0100, Jordi Prats wrote:
>> Hi all,
>> I found on a webpage this two sentences:
>>
>> - The combination of 32-bit CPU and Linux kernel version 2.6.x, the
>> limit of logical volume size is maximized at 16TB.
>>
>> - For Linux kernel 2.6.x running on 64-bit CPU, the maximum LV size is
>> 8EB (extremely terrible big storage for this time being!)
>>
>> Are they right?
> 
> Ok.  But it's really impractical to have large multi-terabyte
> single filesystem today.  What are you wanting to do?  Ever fsck a
> 2TB filesystem?  Consider yourself warned.
> 
> Or... were you just curious about the maximums?
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 

Re: [linux-lvm] LVM limits?

[Jordi Prats]

Agree. But, at least in my case, I've been studying better alternatives
(this means testing) but I've been required to grow the file system
because of our needs.

So, It's not the best solution, but until I can move to a better one
(this probably means buy more hardware) this is my best option to
provide space.

JFS, XFS and other file systems seems ok, but it wouldn't be a better
solution something like Lustre, Ceph or GlusterFS? They seems to provide
better scalability been also POSIX-compliant. What do you think?

Any inputs will be welcome :)

regards,
Jordi

Joseph L. Casale wrote:
>> It took less then 90 minutes.
>>
>> Ehud.
> 
> 90 Minutes, how "enterprisable" is that? Just cause you can, doesn't mean you should: What if that server tanked during working hours at the corp? You would have to sit and wait for it to come back up. Worse than that are the rebuild times for degraded arrays. Try to rebuild a **huge** array on **huge** discs during production with lots of IO and hope you don't lose another disc. There are practical alternatives with thoughtful management to circumvent the need for and 8EB or larger array, but YMMV :)
> 
> jlc
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 

Re: [linux-lvm] LVM limits?

[Chris Cox]

On Tue, 2008-01-29 at 01:38 +0200, Ehud Karni wrote:
> On Mon, 28 Jan 2008 14:52:20 Joseph L. Casale wrote:
> >
> > >It took less then 90 minutes.
> >
> > 90 Minutes, how "enterprisable" is that? Just cause you can, doesn't
> > mean you should: What if that server tanked during working hours at
> > the corp? You would have to sit and wait for it to come back up.
> > Worse than that are the rebuild times for degraded arrays. Try to
> > rebuild a **huge** array on **huge** discs during production with
> > lots of IO and hope you don't lose another disc. There are practical
> > alternatives with thoughtful management to circumvent the need for
> > and 8EB or larger array, but YMMV :)
> 
> First, that's on my home machine (the whole machine, including the
> disks and controller, cost me about $1100).
> 
> At work I have 1.75 TB VG (RAID-5 using 3ware, 320 GB SATA-2 x 7, with
> 1 spare), but it is split into 3 LVs the largest of them is 700 GB.
> I had several rebuilds, It takes about 14 hours with no disturbance
> at all to the work.
> 
> The fsck problem can be dealt in various ways. Changing the FS may be
> one of them but I have no experience in this way. Another, more ready
> way is to tunefs the FSs (all ext3 in my case) so it will never occurs
> automatically, and do scheduled fsck once or twice a year.

Fine... let's forget fsck (brick wall... beating head... etc.).

Let's say you want to make a copy of your 5TB filesystem... how long
does that take?

My point (washed away in silly talk) is that operations on large
filesytems can take a VERY long time.  Just looking at the (very)
trivial examples and not looking that the problem at a whole doesn't
solve the problem (as much as we'd like to think that it does).

> 
> I think that when we'll have much larger arrays, their speed will also
> be larger. For those who interested in ancient history, I recall that
> 12-13 years ago, I had a Clariion at work with just 40 GB (it was 10
> scsi disks of 4GB) and the fsck (on Aviion DGIX) took more then 50
> minutes. Morale - the advancement of technology work for us.

No.  The performance of arrays is not keeping in step with their
growth.  Not even close.  Granted read/write laser holographic
storage may be a solution, but I doubt we'll see anything in
the traditional "disk" storage medium performance that will catch
us up with regards to the amount of space increases.  It's going
to take something fairly radical.

Even distributed filesystems may not be the scalable answer
for this.... though it might help for a little while.

Re: [linux-lvm] LVM limits?

[Michael Eisenkölbl]

Hi,

do you also use a clustered-LVM in your storage system.
i mean, two storages mirrored in a cluster (cmirror ?).

kind regards
Michael

Jordi Prats schrieb:
> Hi,
> I'm the system administrator of PADICAT (http://www.padi.cat). It
> collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to
> provide permanent access to them (http://www.padi.cat/en/quees.php).
> It's equivalent to Internet Archive (http://www.archive.org) but for a
> particular culture.
>
> Our software developers require us to have one large file system,
> actually a single directory, with all this historically-classified web
> sites on a gziped file.
>
> I'm currently studying lustre and other HPC-related file systems to get
> this large file system, but by now I have ext3 as our file system. Next
> Monday I'm planning to extend it to 3TB o 4TB, so I'm currently
> researching for restrictions because during next month I'll have between
> 3TB to 4TB more to add: so, it will become a 8TB file system.
>
> Last time I fsck my 2'1TB file system I spend about 2 hours. Anyway, I'm
> also curious about the maximums :P
>
> Thanks,
> Jordi
>
>
> Chris Cox wrote:
>   
>> On Mon, 2008-01-28 at 20:24 +0100, Jordi Prats wrote:
>>     
>>> Hi all,
>>> I found on a webpage this two sentences:
>>>
>>> - The combination of 32-bit CPU and Linux kernel version 2.6.x, the
>>> limit of logical volume size is maximized at 16TB.
>>>
>>> - For Linux kernel 2.6.x running on 64-bit CPU, the maximum LV size is
>>> 8EB (extremely terrible big storage for this time being!)
>>>
>>> Are they right?
>>>       
>> Ok.  But it's really impractical to have large multi-terabyte
>> single filesystem today.  What are you wanting to do?  Ever fsck a
>> 2TB filesystem?  Consider yourself warned.
>>
>> Or... were you just curious about the maximums?
>>
>> _______________________________________________
>> linux-lvm mailing list
>> linux-lvm@redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-lvm
>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>>
>>
>>     
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>   

Re: [linux-lvm] LVM limits?

[Ehud Karni]

I think this is quiet off topic, so this is my last reply.

On Mon, 28 Jan 2008 18:27:13 Chris Cox wrote:
>
> Let's say you want to make a copy of your 5TB filesystem... how long
> does that take?
>
> My point (washed away in silly talk) is that operations on large
> filesytems can take a VERY long time.  Just looking at the (very)
> trivial examples and not looking that the problem at a whole doesn't
> solve the problem (as much as we'd like to think that it does).

I agree to your basic point, but what is your solution ?
Even if you have distributed FSs, you still have to back it all,
keep its integrity and manage it somehow. I don't see how it is
a lesser problem.

The basic problem is that the data we hold grows faster than the
software/hardware capabilities (or they are unreasonably priced).

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7966-667  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: [linux-lvm] LVM limits?

[Chris Cox]

On Tue, 2008-01-29 at 12:04 +0200, Ehud Karni wrote:
> I think this is quiet off topic, so this is my last reply.
> 
> On Mon, 28 Jan 2008 18:27:13 Chris Cox wrote:
> >
> > Let's say you want to make a copy of your 5TB filesystem... how long
> > does that take?
> >
> > My point (washed away in silly talk) is that operations on large
> > filesytems can take a VERY long time.  Just looking at the (very)
> > trivial examples and not looking that the problem at a whole doesn't
> > solve the problem (as much as we'd like to think that it does).
> 
> I agree to your basic point, but what is your solution ?
> Even if you have distributed FSs, you still have to back it all,
> keep its integrity and manage it somehow. I don't see how it is
> a lesser problem.

The solution... create and use multiple filesystems rather than
one big  one.

> 
> The basic problem is that the data we hold grows faster than the
> software/hardware capabilities (or they are unreasonably priced).

It's growing much, much faster.

Can you imagine if every machine at a 10,000 person shop had
one of those inexpensive terabyte drives... AND somehow they
manage to fill the space up... let's say 50% ... AND then you
have to back it all up?

It's not just a large filesystem problem...

:)

Re: [linux-lvm] LVM limits?

[Lars Ellenberg]

On Tue, Jan 29, 2008 at 12:47:24AM +0100, Jordi Prats wrote:
> Hi,
> I'm the system administrator of PADICAT (http://www.padi.cat). It
> collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to
> provide permanent access to them (http://www.padi.cat/en/quees.php).
> It's equivalent to Internet Archive (http://www.archive.org) but for a
> particular culture.
> 
> Our software developers require us to have one large file system,
> actually a single directory, with all this historically-classified web
> sites on a gziped file.

if I may...
if you are _archiving_ then the archive will be read-only.
so you need a "working area" where you download and prepare the current
stuff, "pack" it into the form of object repository you need.
but then you archive it away into some much larger,
but basically read-only, store.

if that is correct so far,
what about having a some TB volume for the "working area",
and then (for example) cramfs it away into versioned archive images.
cramfs (or other strictly read-only) images have the additional
advantage that they very easily scale out: you just clone it one other
time, done. because of the read-only nature of the archive there is no
need for any coordination, and one clone is as good as the other one.
scales in any direction without effort.

> I'm currently studying lustre and other HPC-related file systems to get
> this large file system, but by now I have ext3 as our file system. Next
> Monday I'm planning to extend it to 3TB o 4TB, so I'm currently
> researching for restrictions because during next month I'll have between
> 3TB to 4TB more to add: so, it will become a 8TB file system.

If you have enterprise class hardware (enough RAM, good battery-backed
controller etc), I personaly do very much prefer XFS.
you should upgrade the "xfsprogs", though,
most distributions ship outdated xfs tools.

<shameless plug>
btw., if you want to explore lustre further, we have some cooperation
with the lustre people to run lustre on DRBD. if interessted in details,
contact address is below...
</shameless plug>

-- 
: Lars Ellenberg                           http://www.linbit.com :
: DRBD/HA support and consulting             sales at linbit.com :
: LINBIT Information Technologies GmbH      Tel +43-1-8178292-0  :
: Vivenotgasse 48, A-1120 Vienna/Europe     Fax +43-1-8178292-82 :
__
please use the "List-Reply" function of your email client.

Re: [linux-lvm] LVM limits?

[David Robinson]

Jordi Prats wrote:
> Hi,
> I'm the system administrator of PADICAT (http://www.padi.cat). It
> collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to
> provide permanent access to them (http://www.padi.cat/en/quees.php).
> It's equivalent to Internet Archive (http://www.archive.org) but for a
> particular culture.
> 
> Our software developers require us to have one large file system,
> actually a single directory, with all this historically-classified web
> sites on a gziped file.
> 
> I'm currently studying lustre and other HPC-related file systems to get
> this large file system, but by now I have ext3 as our file system. Next
> Monday I'm planning to extend it to 3TB o 4TB, so I'm currently
> researching for restrictions because during next month I'll have between
> 3TB to 4TB more to add: so, it will become a 8TB file system.
> 
> Last time I fsck my 2'1TB file system I spend about 2 hours. Anyway, I'm
> also curious about the maximums :P

The man page for vgcreate talks a little bit about limits:

"If the volume group metadata uses lvm1 format, extents can vary in size 
from 8KB to 16GB and there is a limit of 65534 extents in each logical 
volume. The default of 4 MB leads to a maximum logical volume size of 
around 256GB.
If the volume group metadata uses lvm2 format those restrictions do not 
apply, but having a large number of extents will slow down the tools but 
have no impact on I/O performance to the logical volume."

In short, you're more likely to reach filesystem limits before LVM's. 
EXT3 has a theoretical limit of 32 TB, but 32 GB its not practical. 
Creating an EXT3 filesystem larger than 8 TB is umm, brave - as you have 
noticed the tools (eg. fsck) do not scale well w/ EXT3.

GFS or XFS (or others) may be more suitable, but it depends on your 
requirements.

--Dave

Re: [linux-lvm] LVM limits? OT

[Steeve McCauley]

David Robinson wrote:
> Jordi Prats wrote:
>> Hi,
>> I'm the system administrator of PADICAT (http://www.padi.cat). It
>> collects Catalan (http://en.wikipedia.org/wiki/Catalonia) web sites to
>> provide permanent access to them (http://www.padi.cat/en/quees.php).
>> It's equivalent to Internet Archive (http://www.archive.org) but for a
>> particular culture.
>>
>> Our software developers require us to have one large file system,
>> actually a single directory, with all this historically-classified web
>> sites on a gziped file.
>>
>> I'm currently studying lustre and other HPC-related file systems to get
>> this large file system, but by now I have ext3 as our file system. Next
>> Monday I'm planning to extend it to 3TB o 4TB, so I'm currently
>> researching for restrictions because during next month I'll have between
>> 3TB to 4TB more to add: so, it will become a 8TB file system.
>>
>> Last time I fsck my 2'1TB file system I spend about 2 hours. Anyway, I'm
>> also curious about the maximums :P
> 
> The man page for vgcreate talks a little bit about limits:
> 
> "If the volume group metadata uses lvm1 format, extents can vary in size 
> from 8KB to 16GB and there is a limit of 65534 extents in each logical 
> volume. The default of 4 MB leads to a maximum logical volume size of 
> around 256GB.
> If the volume group metadata uses lvm2 format those restrictions do not 
> apply, but having a large number of extents will slow down the tools but 
> have no impact on I/O performance to the logical volume."
> 
> In short, you're more likely to reach filesystem limits before LVM's. 
> EXT3 has a theoretical limit of 32 TB, but 32 GB its not practical. 
> Creating an EXT3 filesystem larger than 8 TB is umm, brave - as you have 
> noticed the tools (eg. fsck) do not scale well w/ EXT3.

With a 4K block size the theoretical limit is 16TB.  The ext2/3
tools do not work beyond 4TB, or at least they didn't as of a year
or two ago.  I did report the problem to Theodore, not sure if the
fix was ever adopted, but you're absolutely right about the scalability
problems of ext3.  An fsck of a 16TB filesystem was something that
definitely required tea.

> GFS or XFS (or others) may be more suitable, but it depends on your 
> requirements.

Any of the extent based filesystems.  A fixed format filesystem like
ext3 does not scale.