Mirroring swap

Mirroring swap

[Derek Yeung]

Hi,

I'm running RedHat 9 on stock kernel 2.4.20 with software RAID 1.  

I was wondering if it's a good idea to also mirror the swap partition?  
I was reading old posts in the newsgroups that suggest this may not be a 
good idea but i can't seem to find an answer as to why.  i think mirroring 
swap would be a good idea as it would increase redundancy..  Any thoughts?  

If it's a good idea, can anyone suggest how i can do it?  (mirror swap?)  
I don't currently have it set up yet.


Thanks,
/dky

RE: Mirroring swap

[Riley Williams]

Hi Derek.

 > I'm running Red Hat 9 on stock kernel 2.4.20 with software
 > RAID 1.
 >
 > I was wondering if it's a good idea to also mirror the swap
 > partition? I was reading old posts in the newsgroups that
 > suggest this may not be a good idea but I can't seem to find
 > an answer as to why. I think mirroring swap would be a good
 > idea as it would increase redundancy. Any thoughts?  

Personally, I can't see the point. the contents of swap are by
definition short term temporary data with copies held either in
memory or elsewhere on the hard disk subsystem.

My preference is to create multiple swap partitions, one on each
hard drive, as in that situation, the swap subsystem always uses
a swap partition on a different drive to the original copy when
it decides where to swap out to.

Remember that using a raided swap partition means that the swap
code has no idea which drive any particular part of that swap
partition is on, so could result in the swap copy being on the
same failed drive as the original copy.

Best wishes from Riley.
---
 * Nothing as pretty as a smile, nothing as ugly as a frown.

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.491 / Virus Database: 290 - Release Date: 18-Jun-2003

Re: Mirroring swap

[Matti Aarnio]

On Thu, Jun 26, 2003 at 08:07:47AM +0100, Riley Williams wrote:
> Hi Derek.
>  > I'm running Red Hat 9 on stock kernel 2.4.20 with software
>  > RAID 1.
>  >
>  > I was wondering if it's a good idea to also mirror the swap
>  > partition? I was reading old posts in the newsgroups that
>  > suggest this may not be a good idea but I can't seem to find
>  > an answer as to why. I think mirroring swap would be a good
>  > idea as it would increase redundancy. Any thoughts?  
> 
> Personally, I can't see the point. the contents of swap are by
> definition short term temporary data with copies held either in
> memory or elsewhere on the hard disk subsystem.

I do that mirroring routinely.

What Linux sends to swap has _no_ origin in anywhere else in
the system, e.g. shared library loading makes runtime modifications
into loaded program, modifications to RW DATA segment, and
of course the BSS segment, and all of heap, and stack...

Most of the executable will usually stay unmodified, and be
backed by its on-disk version, but never everything.

...
> Remember that using a raided swap partition means that the swap
> code has no idea which drive any particular part of that swap
> partition is on, so could result in the swap copy being on the
> same failed drive as the original copy.

Nope.  swap-code does not see it at all.  All of that is hidden with
under the mirroring code, and it does disconnect faulted drive from
the mirror set.

> Best wishes from Riley.

/Matti Aarnio

RE: Mirroring swap

[Gordon Henderson]

On Thu, 26 Jun 2003, Riley Williams wrote:

> Personally, I can't see the point. the contents of swap are by
> definition short term temporary data with copies held either in
> memory or elsewhere on the hard disk subsystem.

Not strictly true. The whole point of "virual memory" is that the OS has
the right to shift parts of, or an entire programs data segment (and
stack, and other volatile data needed to resume execution of that program
which isn't mirrored elsewhere in memory or the filing system)  onto a
swap device when memory becomes tight. If later, it tries to read that
data back to resume execution of the program and it fails then at best it
will abort the program and at worst it'll just give up and panic. I've had
the latter happen when a swap partition developed a bad sector.

So now with important machines which had mirrored or RAIDed disk systems,
I always put swap on a mirrored or RAID5d device.

Eg.

gordonh @ red: cat /proc/swaps
Filename                        Type            Size    Used    Priority
/dev/md2                        partition       1991672 12512   -1

md2 : active raid5 hdm2[2] hdk2[4] hdi2[1] hdg2[3] hde2[0]
      1991680 blocks level 5, 32k chunk, algorithm 0 [5/5] [UUUUU]

> My preference is to create multiple swap partitions, one on each
> hard drive, as in that situation, the swap subsystem always uses
> a swap partition on a different drive to the original copy when
> it decides where to swap out to.

Linux will round-robbin swap allocation between all swap devices, unless
you give a device a higher priority in the fstab. (or swapon command)

Gordon

RE: Mirroring swap

[Gregory Leblanc]

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

On Thu, 2003-06-26 at 01:30, Gordon Henderson wrote:
[snip]
> So now with important machines which had mirrored or RAIDed disk systems,
> I always put swap on a mirrored or RAID5d device.

Swap on RAID 5 is going to have pretty poor performance, in general. 
Swap needs to be both high bandwidth and low latency, both for reading
and writing.  RAID 5 just isn't that.
	Greg.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: Mirroring swap

[Corey McGuire]

Personally, I think that if a system is so vital that it can't afford a
swap failure, then fill it full of RAM.  One can populate a 32bit system to
the max (4GB's) with quality RAM for less than a grand (less than $500 if
you buy real cheap stuff)

I don't know about Linux, but I know in Windows, swap is only vital because
M$ is too lazy to change their code.  Even in windows XP you need swap.
You can turn off "virtual memmory" but swap is just moved to ram (that's
right, real memory pretending to be fake memory pretending to be real
memory)

In Linux, is there a reason for using swap if you can afford the RAM?

*********** REPLY SEPARATOR  ***********

On 6/26/2003 at 2:23 PM Gregory Leblanc wrote:

>On Thu, 2003-06-26 at 01:30, Gordon Henderson wrote:
>[snip]
>> So now with important machines which had mirrored or RAIDed disk
systems,
>> I always put swap on a mirrored or RAID5d device.
>
>Swap on RAID 5 is going to have pretty poor performance, in general. 
>Swap needs to be both high bandwidth and low latency, both for reading
>and writing.  RAID 5 just isn't that.
>	Greg.
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html




/\/\/\/\/\/\ Nothing is foolproof to a talented fool. /\/\/\/\/\/\

coreyfro@coreyfro.com
http://www.coreyfro.com/
http://stats.distributed.net/rc5-64/psummary.php3?id=196879
ICQ : 3168059

-----BEGIN GEEK CODE BLOCK-----
GCS d--(+) s: a-- C++++$ UBL++>++++ P+ L+ E W+++$ N+ o? K? w++++$>+++++$
O---- !M--- V- PS+++ PE++(--) Y+ PGP- t--- 5(+) !X- R(+) !tv b-(+)
Dl++(++++) D++ G+ e>+++ h++(---) r++>+$ y++*>$ H++++ n---(----) p? !au w+
v- 3+>++ j- G'''' B--- u+++*** f* Quake++++>+++++$
------END GEEK CODE BLOCK------

Home of Geek Code - http://www.geekcode.com/
The Geek Code Decoder Page - http://www.ebb.org/ungeek//

Re: Mirroring swap

[Scott McDermott]

Corey McGuire on Thu 26/06 14:44 -0700:
> In Linux, is there a reason for using swap if you can
> afford the RAM?

Hmm...if you really aren't using some pages, why not have
them paged out during some idle time and have more free
physical memory available immediately without having to wait
for pageout, should demand occur? I don't know if Linux will
ever page out even very old pages if there is no memory
pressure at all, but it probably makes sense to do so in
order to have memory available immediately, should a huge
demand occur.  Then again, should those pages be demanded
again, you would have to wait for pagein.  I wonder what
Linux does?

Re: Mirroring swap

[Corey McGuire]

I'd like to know if Linux can address more than 4GB's of RAM (on a 32 bit processor without using dirty tricks like Intel's 36bit hack) including the swap system.

I am pretty sure that all OS's using 32bit CPU's in a 32bit way can only address 4GB's of RAM, including swap.  Ergo, when RAM gets cheap enough, swap could become obsolete.

I'll probably have to blow out my raid before all is done with me.  Anyone know a way I could occupie 4GB's or more RAM quickly?

*********** REPLY SEPARATOR  ***********

On 6/26/2003 at 9:56 PM Scott McDermott wrote:

>Corey McGuire on Thu 26/06 14:44 -0700:
>> In Linux, is there a reason for using swap if you can
>> afford the RAM?
>
>Hmm...if you really aren't using some pages, why not have
>them paged out during some idle time and have more free
>physical memory available immediately without having to wait
>for pageout, should demand occur? I don't know if Linux will
>ever page out even very old pages if there is no memory
>pressure at all, but it probably makes sense to do so in
>order to have memory available immediately, should a huge
>demand occur.  Then again, should those pages be demanded
>again, you would have to wait for pagein.  I wonder what
>Linux does?
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html




/\/\/\/\/\/\ Nothing is foolproof to a talented fool. /\/\/\/\/\/\

coreyfro@coreyfro.com
http://www.coreyfro.com/
http://stats.distributed.net/rc5-64/psearch.php3?st=coreyfro
ICQ : 3168059

-----BEGIN GEEK CODE BLOCK-----
GCS !d--(+) s: a- C++++$ UL++>++++ P+ L++>++++ E- W+++$ N++ o? K? w++++$>+++++$ O---- !M--- V- PS+++ PE++(--) Y+ PGP- t--- 5(+) !X- R(+) !tv b-(+) Dl++(++++) D++ G++(-) e>+++ h++(---) r++>+$ y++**>$ H++++ n---(----) p? !au w+ v- 3+>++ j- G'''' B--- u+++*** f* Quake++++>+++++$
------END GEEK CODE BLOCK------

Home of Geek Code - http://www.geekcode.com/
The Geek Code Decoder Page - http://www.ebb.org/ungeek//

Re: Mirroring swap

[Scott McDermott]

Corey McGuire on Thu 26/06 19:14 -0700:
> I'd like to know if Linux can address more than 4GB's of
> RAM (on a 32 bit processor without using dirty tricks like
> Intel's 36bit hack) including the swap system.

Without PAE (36-bit), then of course the limit is 32-bit
addressing.

You're right that 4G is not that expensive, but only certain
workloads can actually use this much memory.  If your
working set is only 256M, but you often do other work with
different working sets, this is a good candidate for disk
swap.  Since you only use the one working set at a time, you
really don't need to fit all of your working sets in core at
once and you are just wasting money.  Disk is still MUCH
cheaper than fast memory.

Re: Mirroring swap

[Corey McGuire]

Granted.  And it will be a while before I even bust 512MB's with a personal system (I'm waiting for hammer time!  Then I can have my 4GB's AND swap) but as far as reliability, solid-state beats spinning platters ;-)

And its not like gobs of free RAM is useless in the Linux world.  Unlike windows, Linux keeps it RAM filled with cache until free RAM is needed (I have disk intensive windows apps that run far faster under WINE than Windows.  I love bragging about them ;-) 

Cache or swap, cache or swap.  Cash for cache or crash for swap.

PS, on a RAID related note, 10.7 out of the 12.5% crash point on my RAID5 rebuild. without DMA, 5907.1 minutes before my RAID sync's.

DMA, its what's for dinner.

*********** REPLY SEPARATOR  ***********

On 6/26/2003 at 10:24 PM Scott McDermott wrote:

>Corey McGuire on Thu 26/06 19:14 -0700:
>> I'd like to know if Linux can address more than 4GB's of
>> RAM (on a 32 bit processor without using dirty tricks like
>> Intel's 36bit hack) including the swap system.
>
>Without PAE (36-bit), then of course the limit is 32-bit
>addressing.
>
>You're right that 4G is not that expensive, but only certain
>workloads can actually use this much memory.  If your
>working set is only 256M, but you often do other work with
>different working sets, this is a good candidate for disk
>swap.  Since you only use the one working set at a time, you
>really don't need to fit all of your working sets in core at
>once and you are just wasting money.  Disk is still MUCH
>cheaper than fast memory.
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html




/\/\/\/\/\/\ Nothing is foolproof to a talented fool. /\/\/\/\/\/\

coreyfro@coreyfro.com
http://www.coreyfro.com/
http://stats.distributed.net/rc5-64/psearch.php3?st=coreyfro
ICQ : 3168059

-----BEGIN GEEK CODE BLOCK-----
GCS !d--(+) s: a- C++++$ UL++>++++ P+ L++>++++ E- W+++$ N++ o? K? w++++$>+++++$ O---- !M--- V- PS+++ PE++(--) Y+ PGP- t--- 5(+) !X- R(+) !tv b-(+) Dl++(++++) D++ G++(-) e>+++ h++(---) r++>+$ y++**>$ H++++ n---(----) p? !au w+ v- 3+>++ j- G'''' B--- u+++*** f* Quake++++>+++++$
------END GEEK CODE BLOCK------

Home of Geek Code - http://www.geekcode.com/
The Geek Code Decoder Page - http://www.ebb.org/ungeek//

Re: Mirroring swap

[Ricky Beam]

On Thu, 26 Jun 2003, Corey McGuire wrote:
>And its not like gobs of free RAM is useless in the Linux world.  Unlike
>windows, Linux keeps it RAM filled with cache until free RAM is needed (I
>have disk intensive windows apps that run far faster under WINE than
>Windows.  I love bragging about them ;-)

"Unlike windows"?  Do you even *use* windows?  Windows has had the same
filesystem caching capability as linux for many years now.  Linux is a lot
more lax about what it'll keep in memory -- basically, cache pages will
not be pushed out until something needs the memory (application or another
cache page.)  Windows limits the ammount of cached space and tends to dump
things a great deal faster...

Everyone loves to poopoo on windows without even looking.  Both do caching.
Linux tends to do it much better -- esp. under load; copy a file 2x the
size of RAM on both and you'll see what I mean.

--Ricky

Re: Show me the cache, was Mirroring swap

[Corey McGuire]

>"Unlike windows"?  Do you even *use* windows?  Windows has had the same
>filesystem caching capability as linux for many years now.  Linux is a lot
>more lax about what it'll keep in memory -- basically, cache pages will
>not be pushed out until something needs the memory (application or another
>cache page.)  Windows limits the ammount of cached space and tends to dump
>things a great deal faster...

Dude, that is what I said.

What I said was windows gives up way too quickly.  What I did not say is
that it doesn't cache at all, which would be insane.  Windows NT's memory
manglement is "pretty good" apart from its forgetfulness.

What is most annoying is there are hooks for tweaking cache size, but even
then, windows doesn't listen.  The only options in the windows registry
that have any lasting effects are these.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory
Management]
"DisablePagingExecutive"=dword:00000001
"LargeSystemCache"=dword:00000001

The first one doesn't even effect cache, it just doesn't swap the kernel,
which greatly reduces the load on the cache.  "LargeSystemCache" hardly
helps, as it basicly tells Windows "yes, all RAM _CAN_ be used for cache"
instead of just a paultry 4MB.  Does this help?  Yeah, my 512MB machine is
currently using 18MB's of cache.

There are cache replacements on the web, but most are overpriced for what
they do.  Any free cache program is bogus (like system internal's cacheset)
because they use system calls that are most often ignored and frequently
reset, at least, from what I have seen in my attempts to make Windows NT
cache intelligently.



/\/\/\/\/\/\ Nothing is foolproof to a talented fool. /\/\/\/\/\/\

coreyfro@coreyfro.com
http://www.coreyfro.com/
http://stats.distributed.net/rc5-64/psummary.php3?id=196879
ICQ : 3168059

-----BEGIN GEEK CODE BLOCK-----
GCS d--(+) s: a-- C++++$ UBL++>++++ P+ L+ E W+++$ N+ o? K? w++++$>+++++$
O---- !M--- V- PS+++ PE++(--) Y+ PGP- t--- 5(+) !X- R(+) !tv b-(+)
Dl++(++++) D++ G+ e>+++ h++(---) r++>+$ y++*>$ H++++ n---(----) p? !au w+
v- 3+>++ j- G'''' B--- u+++*** f* Quake++++>+++++$
------END GEEK CODE BLOCK------

Home of Geek Code - http://www.geekcode.com/
The Geek Code Decoder Page - http://www.ebb.org/ungeek//

Re: Show me the cache, was Mirroring swap

[Ricky Beam]

On Fri, 27 Jun 2003, Corey McGuire wrote:
>"DisablePagingExecutive"=dword:00000001

(That doesn't actually work.  NT/2000/XP will still page the kernel parts.
 But not much of it.)

>...  "LargeSystemCache" hardly
>helps, as it basicly tells Windows "yes, all RAM _CAN_ be used for cache"
>instead of just a paultry 4MB.  Does this help?  Yeah, my 512MB machine is
>currently using 18MB's of cache.

Currently, my XP Home laptop is using 356,188K of cache.  That will go to
"all available RAM" if an application starts writing large volumes of data
to disk. (Linux throttles processes that flood the cache.)

--Ricky

Re: Show me the cache, was Mirroring swap

[Corey McGuire]

Understand, but my point was that my disk happy windows apps actually work
better under wine than windows due to more persistent caching.

I have this asset-tracking app that I use to process application licenses.
It keeps all its data in a big text file.  In windows, processing the
results from a single system can take up to a minute.  In Linux, this takes
seconds.

Wizardry 8, a fun RPG, takes huge amounts of time to load new maps under
windows.  In linux, these maps load much more quickly.

Win-mx generates "bit prints" for files so when you search for a file, as
long as you know its bit print (actually, you don't need to, the program
handles it) you can easily find THAT SPECIFIC FILE on other peoples boxes.
Very nice!  Generating these bit prints is also faster under linux, but
this probably has less to do with cache and more to do with driver
performance.

*********** REPLY SEPARATOR  ***********

On 6/27/2003 at 2:24 PM Ricky Beam wrote:

>On Fri, 27 Jun 2003, Corey McGuire wrote:
>>"DisablePagingExecutive"=dword:00000001
>
>(That doesn't actually work.  NT/2000/XP will still page the kernel parts.
> But not much of it.)
>
>>...  "LargeSystemCache" hardly
>>helps, as it basicly tells Windows "yes, all RAM _CAN_ be used for cache"
>>instead of just a paultry 4MB.  Does this help?  Yeah, my 512MB machine
is
>>currently using 18MB's of cache.
>
>Currently, my XP Home laptop is using 356,188K of cache.  That will go to
>"all available RAM" if an application starts writing large volumes of data
>to disk. (Linux throttles processes that flood the cache.)
>
>--Ricky
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html




/\/\/\/\/\/\ Nothing is foolproof to a talented fool. /\/\/\/\/\/\

coreyfro@coreyfro.com
http://www.coreyfro.com/
http://stats.distributed.net/rc5-64/psummary.php3?id=196879
ICQ : 3168059

-----BEGIN GEEK CODE BLOCK-----
GCS d--(+) s: a-- C++++$ UBL++>++++ P+ L+ E W+++$ N+ o? K? w++++$>+++++$
O---- !M--- V- PS+++ PE++(--) Y+ PGP- t--- 5(+) !X- R(+) !tv b-(+)
Dl++(++++) D++ G+ e>+++ h++(---) r++>+$ y++*>$ H++++ n---(----) p? !au w+
v- 3+>++ j- G'''' B--- u+++*** f* Quake++++>+++++$
------END GEEK CODE BLOCK------

Home of Geek Code - http://www.geekcode.com/
The Geek Code Decoder Page - http://www.ebb.org/ungeek//

RE: Mirroring swap

[Gordon Henderson]

On 26 Jun 2003, Gregory Leblanc wrote:

> On Thu, 2003-06-26 at 01:30, Gordon Henderson wrote:
> [snip]
> > So now with important machines which had mirrored or RAIDed disk systems,
> > I always put swap on a mirrored or RAID5d device.
>
> Swap on RAID 5 is going to have pretty poor performance, in general.
> Swap needs to be both high bandwidth and low latency, both for reading
> and writing.  RAID 5 just isn't that.

I don't actually care right now - these machines are servers and I really
don't want them to swap - I'd rather run them without swap at all, but
I've yet to see something that'll convince me that this is a good idea,
and anyway.. Bonnie might not be the best thing in disk benchmarks, but..

Version 1.02b       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
red              1G 15418  99 68565  37 31359  24 14286  97 150174  84 390.6   1

is fairly reasonable.

G

Re: Mirroring swap

[Scott McDermott]

Derek Yeung on Wed 25/06 15:39 -0400:
> I was wondering if it's a good idea to also mirror the
> swap partition?  

You should definitely mirror swap or any dirty pages which
are swap-backed will be lost when the swap device dies,
resulting in data or filesystem corruption.  Not to mention
that your system will almost certainly crash when this
occurs.

> I was reading old posts in the newsgroups that suggest
> this may not be a good idea but i can't seem to find an
> answer as to why.

The old RAID code couldn't handle swap for some reason, but
that was fixed a long time ago.  I believe it had something
to do with deadlocking in a situation where the RAID code
needed memory for some dealings with the mirrored swap
device, but the memory couldn't be obtained without swapping
out, which as you can imagine would cause problems.  I think
they changed this by making the kinds of buffers used by
RAID in these situations to be pinned, but I don't know.

> If it's a good idea, can anyone suggest how i can do it?
> (mirror swap?)  I don't currently have it set up yet.

swapoff, mkraid, raidstart, mkswap, swapon, vi /etc/fstab

RE: Mirroring swap

[Bailey, Scott]

>I was wondering if it's a good idea to also mirror the swap partition?  

I've heard both sides of the argument, but my 2 cents says "mirror it".
Otherwise you run the risk of having your swap area become unusable
following a drive failure, which will make your system very upset if it
decides it needs to swap in (or out) data associated with that area.

>If it's a good idea, can anyone suggest how i can do it?  (mirror swap?)  

It's just another partition. :-) Assuming you have created a md device of
the appropriate size, just use "/dev/md5" [or whatever] in your mkswap
command and the first column of the swap entry in your /etc/fstab. Note that
you do *NOT* have separate partitions on the md device itself.

My preference in this situation actually is to use LVM on top of RAID so
that you don't have to mess around with so many md devices. In this
scenario, you create a much larger md device, put it into a volume group,
then create a logical volume of the desired size. That is: (very
abbreviated)

# pvcreate /dev/md5
# vgcreate myvg /dev/md5
# lvcreate --contiguous y --size 128m --name swapvol myvg
# mkswap /dev/myvg/swapvol
# swapon /dev/myvg/swapvol

I can't remember making the volume contiguous is actually a requirement for
Linux LVM or just garbage cluttering up my brain from HP-UX days. :-) 

Have fun,

	Scott