Swap maximum size documented ?

Swap maximum size documented ?

[david.balazic]

Hi! 

Is there any doc about swap size limits ? 

The mkwap(8) man page claims, that currently the limit is 
32 swap areas of maximum 2 gigabyte size (for x86 arch). 

Is that correct ? 

Regards, 

David Balazic

Re: Swap maximum size documented ?

[William Lee Irwin III]

On Wed, Jun 01, 2005 at 02:25:13PM +0200, david.balazic@hermes.si wrote:
> Is there any doc about swap size limits ? 
> The mkwap(8) man page claims, that currently the limit is 
> 32 swap areas of maximum 2 gigabyte size (for x86 arch). 
> Is that correct ? 

No.

$ swapon -s
Filename                                Type            Size    Used    Priority
/dev/hda6                               partition       4000144 0       -1


-- wli

Re: Swap maximum size documented ?

[Jakob Oestergaard]

On Wed, Jun 01, 2005 at 02:25:13PM +0200, david.balazic@hermes.si wrote:
> Hi! 
> 
> Is there any doc about swap size limits ? 

Documetation?  Is this a trick question?  It's Linux, of course there is
no current documentation except for the source  ;)

 /me ducks and runs   ;)

> The mkwap(8) man page claims, that currently the limit is 
> 32 swap areas of maximum 2 gigabyte size (for x86 arch). 
> 
> Is that correct ? 

Not on 2.6 kernels, no.

[sparrow:joe] $ cat /proc/swaps
Filename                 Type            Size    Used Priority
/dev/mapper/vg0-swap     partition       8388600 0    -1

I use 4-8 G swap areas on 32-bit x86 and 64-bit amd64 kernels.

You probably need a new version of mkswap if it insists that 2G is the
maximum - it sure isn't a kernel limitation anymore.

-- 

 / jakob

Re: Swap maximum size documented ?

[Arjan van de Ven]

> > The mkwap(8) man page claims, that currently the limit is 
> > 32 swap areas of maximum 2 gigabyte size (for x86 arch). 
> > 
> > Is that correct ? 
> 
> Not on 2.6 kernels, no.

it's not even true for 2.4 kernels btw; it was a 2.2 and before issue

Re: Swap maximum size documented ?

[David Balažic]

Arjan van de Ven <arjan <at> infradead.org> writes:

> 
> 
> > > The mkwap(8) man page claims, that currently the limit is 
> > > 32 swap areas of maximum 2 gigabyte size (for x86 arch). 
> > > 
> > > Is that correct ? 
> > 
> > Not on 2.6 kernels, no.
> 
> it's not even true for 2.4 kernels btw; it was a 2.2 and before issue

OK, so can anyone tell the actual, current limits ?

Regards,
David

Re: Swap maximum size documented ?

[William Lee Irwin III]

On Wed, Jun 01, 2005 at 01:02:13PM +0000, David Bala??ic wrote:
> OK, so can anyone tell the actual, current limits ?

Without CONFIG_HIGHMEM64G=y you have:
32 swapfiles, max swapfile size of 64GB.

With CONFIG_HIGHMEM64G=y you have:
32 swapfiles, max swapfile size of 512GB.


-- wli

Re: Swap maximum size documented ?

[Bill Davidsen]

William Lee Irwin III wrote:
> On Wed, Jun 01, 2005 at 01:02:13PM +0000, David Bala??ic wrote:
> 
>>OK, so can anyone tell the actual, current limits ?
> 
> 
> Without CONFIG_HIGHMEM64G=y you have:
> 32 swapfiles, max swapfile size of 64GB.
> 
> With CONFIG_HIGHMEM64G=y you have:
> 32 swapfiles, max swapfile size of 512GB.

Does this apply to mmap as well? I have an application which currently 
uses 9TB of data, and one thought to boost performance was to mmap the 
data. Unfortunately, I know 16TB isn't going to be enough for more than 
a few more years :-(
-- 
bill davidsen <davidsen@tmr.com>
   CTO TMR Associates, Inc
   Doing interesting things with small computers since 1979

Re: Swap maximum size documented ?

[William Lee Irwin III]

William Lee Irwin III wrote:
>> Without CONFIG_HIGHMEM64G=y you have:
>> 32 swapfiles, max swapfile size of 64GB.
>> With CONFIG_HIGHMEM64G=y you have:
>> 32 swapfiles, max swapfile size of 512GB.

On Wed, Jun 01, 2005 at 03:10:59PM -0400, Bill Davidsen wrote:
> Does this apply to mmap as well? I have an application which currently 
> uses 9TB of data, and one thought to boost performance was to mmap the 
> data. Unfortunately, I know 16TB isn't going to be enough for more than 
> a few more years :-(

This only applies to swapping on ia32/i386.

mmap() is limited only by file offsets, which are fully 32-bit on
32-bit systems. remap_file_pages() is limited by PTE_FILE_MAX_BITS,
which is fully 32-bit with CONFIG_HIGHMEM64G=y on i386 but only 29 bit
without it on i386. In general checking for PTE_FILE_MAX_BITS on the
relevant architecture should answer your question for remap_file_pages(),
and BITS_PER_LONG for mmap(). The swap limits for other architectures
will also differ and you generally have to look at the swp_entry/pte
encoding/decoding macros to decipher what the precise limits are
(though a quick hacky C program can help discover them for you).
Generally you get the filesizes by PAGE_SIZE << X_FILE_OFFSET_BITS.

It is in principle possible to sweep the kernel to allow larger file
offsets on 32-bit systems (pgoff_t is something of a preparation for
that), but I wouldn't advise trying it without rather strong kernel-fu
and much willingness to debug it by one's self, and that with a common
failure mode of fs data corruption. Widening swp_entry_t is slightly
harder as the ptes have limited capacity so you have to somehow allocate
extra data in a deadlock-free manner, but one also has less disruptive
failure modes. I suspect you're not itching to implement these things.

One thing to keep in mind is that these are only permissible filesizes.
Virtualspace must be managed properly for windowing where it's limited
and to prevent pagetable proliferation where it's not.


-- wli

Re: Swap maximum size documented ?

[Bill Davidsen]

William Lee Irwin III wrote:

>William Lee Irwin III wrote:
>  
>
>>>Without CONFIG_HIGHMEM64G=y you have:
>>>32 swapfiles, max swapfile size of 64GB.
>>>With CONFIG_HIGHMEM64G=y you have:
>>>32 swapfiles, max swapfile size of 512GB.
>>>      
>>>
>
>On Wed, Jun 01, 2005 at 03:10:59PM -0400, Bill Davidsen wrote:
>  
>
>>Does this apply to mmap as well? I have an application which currently 
>>uses 9TB of data, and one thought to boost performance was to mmap the 
>>data. Unfortunately, I know 16TB isn't going to be enough for more than 
>>a few more years :-(
>>    
>>
>
>This only applies to swapping on ia32/i386.
>
>mmap() is limited only by file offsets, which are fully 32-bit on
>32-bit systems. remap_file_pages() is limited by PTE_FILE_MAX_BITS,
>which is fully 32-bit with CONFIG_HIGHMEM64G=y on i386 but only 29 bit
>without it on i386. In general checking for PTE_FILE_MAX_BITS on the
>relevant architecture should answer your question for remap_file_pages(),
>and BITS_PER_LONG for mmap(). The swap limits for other architectures
>will also differ and you generally have to look at the swp_entry/pte
>encoding/decoding macros to decipher what the precise limits are
>(though a quick hacky C program can help discover them for you).
>Generally you get the filesizes by PAGE_SIZE << X_FILE_OFFSET_BITS.
>
>It is in principle possible to sweep the kernel to allow larger file
>offsets on 32-bit systems (pgoff_t is something of a preparation for
>that), but I wouldn't advise trying it without rather strong kernel-fu
>and much willingness to debug it by one's self, and that with a common
>failure mode of fs data corruption. Widening swp_entry_t is slightly
>harder as the ptes have limited capacity so you have to somehow allocate
>extra data in a deadlock-free manner, but one also has less disruptive
>failure modes. I suspect you're not itching to implement these things.
>
>One thing to keep in mind is that these are only permissible filesizes.
>Virtualspace must be managed properly for windowing where it's limited
>and to prevent pagetable proliferation where it's not.
>

Thank you for taking the time to give me such a complete reply (I saved 
it to prevent asking similar in the future). Hopefully I will be able to 
leave the problem to someone else by then.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Swap maximum size documented ?

[William Lee Irwin III]

On Wed, Jun 01, 2005 at 03:47:05PM -0400, Bill Davidsen wrote:
>> mmap() is limited only by file offsets, which are fully 32-bit on
>> 32-bit systems. remap_file_pages() is limited by PTE_FILE_MAX_BITS,
>> which is fully 32-bit with CONFIG_HIGHMEM64G=y on i386 but only 29 bit
>> without it on i386. In general checking for PTE_FILE_MAX_BITS on the
>> relevant architecture should answer your question for remap_file_pages(),
>> and BITS_PER_LONG for mmap(). The swap limits for other architectures
>> will also differ and you generally have to look at the swp_entry/pte
>> encoding/decoding macros to decipher what the precise limits are
>> (though a quick hacky C program can help discover them for you).
>> Generally you get the filesizes by PAGE_SIZE << X_FILE_OFFSET_BITS.

The hacky C program. Note that some fiddling with compiler flags is
needed to get it to compile at all, and you probably need to have
done make oldconfig or similar to prep the kernel tree too. The answer
necessarily varies with your .config and it won't work cross-arch
except in the 32/64 -bit variant case (in which case it must be
compiled as the same kind of app [32/64] the kernel is expected to be).

The expressions for the swap limits were taken from sys_swapon().

This should remove all doubt, if any remain.


#include <linux/config.h>
#include <linux/swap.h>
#include <asm/pgtable.h>
#include <linux/swapops.h>

int printf(const char *, ...);

int main(void)
{
	unsigned long long type, offset;

	type = swp_type(pte_to_swp_entry(swp_entry_to_pte(swp_entry(~0UL,0))));
	offset = swp_offset(pte_to_swp_entry(swp_entry_to_pte(swp_entry(0,~0UL)))) - 1;
	offset = (offset << PAGE_SHIFT) + ~PAGE_MASK;

	printf("max swapfiles = %Lu files\n", type);
	printf("max swapsize = %Lu B\n", offset);
	return 0;
}

Re: Swap maximum size documented ?

[cutaway]

----- Original Message ----- 
From: "Bill Davidsen" <davidsen@tmr.com>
To: "William Lee Irwin III" <wli@holomorphy.com>
Cc: <linux-kernel@vger.kernel.org>
Sent: Wednesday, June 01, 2005 15:47
Subject: Re: Swap maximum size documented ?


> >
> >>Does this apply to mmap as well? I have an application which currently
> >>uses 9TB of data....

With this much data, you might consider investigating its compressability.
If it turned out to be highly compressable, algorithmic changes in how the
data is handled might considerably lighten the load on MM and disk
subsystems.  Its almost always cheaper to compress/decompress cached data in
memory than hit physical media.

Re: Swap maximum size documented ?

[Bill Davidsen]

cutaway@bellsouth.net wrote:

>----- Original Message ----- 
>From: "Bill Davidsen" <davidsen@tmr.com>
>To: "William Lee Irwin III" <wli@holomorphy.com>
>Cc: <linux-kernel@vger.kernel.org>
>Sent: Wednesday, June 01, 2005 15:47
>Subject: Re: Swap maximum size documented ?
>
>
>  
>
>>>>Does this apply to mmap as well? I have an application which currently
>>>>uses 9TB of data....
>>>>        
>>>>
>
>With this much data, you might consider investigating its compressability.
>If it turned out to be highly compressable, algorithmic changes in how the
>data is handled might considerably lighten the load on MM and disk
>subsystems.  Its almost always cheaper to compress/decompress cached data in
>memory than hit physical media.
>

The majority of the records are compressed as of about a decade ago.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Swap maximum size documented ?

[Lennart Sorensen]

On Wed, Jun 01, 2005 at 03:10:59PM -0400, Bill Davidsen wrote:
> Does this apply to mmap as well? I have an application which currently 
> uses 9TB of data, and one thought to boost performance was to mmap the 
> data. Unfortunately, I know 16TB isn't going to be enough for more than 
> a few more years :-(

Just buy an Opteron/Athlon64 system and you should be able to mmap it
just fine.  At least if you run an x86_64/amd64 kernel.

Len Sorensen

Re: Swap maximum size documented ?

[William Lee Irwin III]

On Wed, Jun 01, 2005 at 03:10:59PM -0400, Bill Davidsen wrote:
>> Does this apply to mmap as well? I have an application which currently 
>> uses 9TB of data, and one thought to boost performance was to mmap the 
>> data. Unfortunately, I know 16TB isn't going to be enough for more than 
>> a few more years :-(

On Wed, Jun 01, 2005 at 04:43:50PM -0400, Lennart Sorensen wrote:
> Just buy an Opteron/Athlon64 system and you should be able to mmap it
> just fine.  At least if you run an x86_64/amd64 kernel.

Linux does not entail subscription to hardware upgrade treadmills. No
one should be forced by "peer pressure" or Linux deficiencies to buy
new hardware. And this is the single greatest thing about Linux ever.

O_LARGEFILE and current mmap() code will save him the cost of newer
hardware for the near term, and should he be particularly strapped for
cash later on when more than 16TB is needed, he knows to look at making
pgoff_t and/or swp_entry_t 64-bit on his own. There is no need for new
hardware, merely a choice between money and programming effort should
he break the 16TB barrier.


-- wli

Re: Swap maximum size documented ?

[Lennart Sorensen]

On Wed, Jun 01, 2005 at 01:54:51PM -0700, William Lee Irwin III wrote:
> Linux does not entail subscription to hardware upgrade treadmills. No
> one should be forced by "peer pressure" or Linux deficiencies to buy
> new hardware. And this is the single greatest thing about Linux ever.

Sure, I still use a 486 with linux for some jobs, but I am realistic
about what I can expect from that level of hardware.

> O_LARGEFILE and current mmap() code will save him the cost of newer
> hardware for the near term, and should he be particularly strapped for
> cash later on when more than 16TB is needed, he knows to look at making
> pgoff_t and/or swp_entry_t 64-bit on his own. There is no need for new
> hardware, merely a choice between money and programming effort should
> he break the 16TB barrier.

True, although I would think anyone doing actual work on that kind of
data size would be using fairly new hardware anyhow.

If you have to write all sorts of complex algorithms to work around a
limitation in your current hardware, perhaps it is cheaper to buy newer
hardware without that limitation.  Ofcourse if you are working on things
in your spare time for free, then hardware upgrades are always the most
expensive solution.

Len Sorensen

Re: Swap maximum size documented ?

[Bill Davidsen]

Lennart Sorensen wrote:

>On Wed, Jun 01, 2005 at 01:54:51PM -0700, William Lee Irwin III wrote:
>  
>
>>Linux does not entail subscription to hardware upgrade treadmills. No
>>one should be forced by "peer pressure" or Linux deficiencies to buy
>>new hardware. And this is the single greatest thing about Linux ever.
>>    
>>
>
>Sure, I still use a 486 with linux for some jobs, but I am realistic
>about what I can expect from that level of hardware.
>
>  
>
>>O_LARGEFILE and current mmap() code will save him the cost of newer
>>hardware for the near term, and should he be particularly strapped for
>>cash later on when more than 16TB is needed, he knows to look at making
>>pgoff_t and/or swp_entry_t 64-bit on his own. There is no need for new
>>hardware, merely a choice between money and programming effort should
>>he break the 16TB barrier.
>>    
>>
>
>True, although I would think anyone doing actual work on that kind of
>data size would be using fairly new hardware anyhow.
>
>If you have to write all sorts of complex algorithms to work around a
>limitation in your current hardware, perhaps it is cheaper to buy newer
>hardware without that limitation.  Ofcourse if you are working on things
>in your spare time for free, then hardware upgrades are always the most
>expensive solution.
>

I suspect that the company would like to depreciate 30+ 4-way Xeon 
rackmount systems over a little longer than two years. The actual 
limiting factor is probably not money but getting the corporate 
configuration committee to go 64 bit...

Thanks for all the input.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979