Slow swapon for big (12GB) swap

Slow swapon for big (12GB) swap

[Grzegorz Kulewski]

Hi,

I am using big swap here (as a backing for potentially huge tmpfs). And I 
wonder why swapon on such big (like 12GB) swap takes about 7 minutes 
(continuous disk IO). Is this expected? Why it is like that? Can I do 
anything to speed it up? Or maybe remove it into the background with low 
priority or something like that?


Thanks in advance,

Grzegorz Kulewski

Re: Slow swapon for big (12GB) swap

[Helge Hafting]

Grzegorz Kulewski wrote:

> Hi,
>
> I am using big swap here (as a backing for potentially huge tmpfs). 
> And I wonder why swapon on such big (like 12GB) swap takes about 7 
> minutes (continuous disk IO). Is this expected? Why it is like that? 
> Can I do anything to speed it up? Or maybe remove it into the 
> background with low priority or something like that?

I don't know why it is slow.  But you can certainly do something like:

nice swapon /dev/yourdisk &

Then it will happen in the background and with low priority.  Of course,
you can't start filling your tmpfs until this completes.


I don't think tmpfs+swap was made with this sort of use in mind,
so you may want to test the performance when you fill up such a
tmpfs, and compare to the performance of /tmp on a 12GB
ordinary filesystem.  It seems to me that the advantage of /tmp on
tmpfs is lost completely if most of it has to be written to disk anyway.
(Ordinary filesystems are cached too, the "tmpfs advantage" is that
truly temporary (but possibly long-lived) files are never written
to disk _if_ you have enough memory.  /tmp on a plain filesystem
is just as fast due to caching, but may delay other use of the
disk as the ordinary filesystem writes stuff out so it will be
saved for the future.)

If you go for a plain filesystem, consider ext2 which is simple faster
than journalling systems like ext3.  You don't need the added
safety for temporary stuff.  Now ext2 have long fsck times if
something goes wrong, but you don't need to fsck this filesystem.
Have your bootscripts run mke2fs at boot instead of fsck for
this filesystem.  mke2fs on 12GB is fast, much faster than
7 minutes. Expect a few seconds only.

Helge Hafting

Re: Slow swapon for big (12GB) swap

[Andrew Morton]

Grzegorz Kulewski <kangur@polcom.net> wrote:
>
> I am using big swap here (as a backing for potentially huge tmpfs). And I 
>  wonder why swapon on such big (like 12GB) swap takes about 7 minutes 
>  (continuous disk IO).

It's a bit quicker here:

vmm:/usr/src/25# mkswap /dev/hda6
Setting up swapspace version 1, size = 54031826 kB
vmm:/usr/src/25# time swapon /dev/hda6
swapon /dev/hda6  0.00s user 0.04s system 74% cpu 0.054 total


> Is this expected?

Nope.

> Why it is like that?

Are you using a swapfile or a swap partition?

If it's a swapfile then perhaps the filesystem is being inefficient in its
bmap() function.  Which filesystem is it?

Re: Slow swapon for big (12GB) swap

[Antonio Vargas]

On 4/10/06, Andrew Morton <akpm@osdl.org> wrote:
> Grzegorz Kulewski <kangur@polcom.net> wrote:
> >
> > I am using big swap here (as a backing for potentially huge tmpfs). And I
> >  wonder why swapon on such big (like 12GB) swap takes about 7 minutes
> >  (continuous disk IO).
>
> It's a bit quicker here:
>
> vmm:/usr/src/25# mkswap /dev/hda6
> Setting up swapspace version 1, size = 54031826 kB
> vmm:/usr/src/25# time swapon /dev/hda6
> swapon /dev/hda6  0.00s user 0.04s system 74% cpu 0.054 total
>
>
> > Is this expected?
>
> Nope.
>

I'm running swap on a 8G LVM logical volume and /tmp on tmpfs and it
works great (used for compilation & DVD-burning stagning area from
other hosts on the network), with no delays on swapon nor mount.


--
Greetz, Antonio Vargas aka winden of network

http://wind.codepixel.com/
windNOenSPAMntw@gmail.com
thesameasabove@amigascne.org

Every day, every year
you have to work
you have to study
you have to scene.

Re: Slow swapon for big (12GB) swap

[Andrew Morton]

Grzegorz Kulewski <kangur@polcom.net> wrote:
>
> On Mon, 10 Apr 2006, Andrew Morton wrote:
> > Grzegorz Kulewski <kangur@polcom.net> wrote:
> >>
> >> I am using big swap here (as a backing for potentially huge tmpfs). And I
> >>  wonder why swapon on such big (like 12GB) swap takes about 7 minutes
> >>  (continuous disk IO).
> >
> > It's a bit quicker here:
> >
> > vmm:/usr/src/25# mkswap /dev/hda6
> > Setting up swapspace version 1, size = 54031826 kB
> > vmm:/usr/src/25# time swapon /dev/hda6
> > swapon /dev/hda6  0.00s user 0.04s system 74% cpu 0.054 total
> >
> >
> >> Is this expected?
> >
> > Nope.
> >
> >> Why it is like that?
> >
> > Are you using a swapfile or a swap partition?
> 
> Swap file.
> 
> 
> > If it's a swapfile then perhaps the filesystem is being inefficient in its
> > bmap() function.  Which filesystem is it?
> 
> Ext3.
> 

OK.  A 12G file is around 3000 indirect blocks.  At 10 milliseconds each
I'd expect it to take 30-odd seconds.  A 14G swapfile here takes 20 second
for swapon.

Maybe your disks are slow, or the fs is already quite full+fragmented.

I guess the only thing I can suggest is that you force the swapfile to put
its indirect blocks close together.  You can do that by writing one byte
at each 4MB interval:

	creat(swapfile);
	for (i = 0; i < swapfile_size / 4MB; i++)
		pwrite(fd, "\0", 1, i * 4MB);

then fill in all the holes:

  dd if=/dev/zero of=swapfile bs=4MB count=swapfile_size/4MB conv=notrunc

rather kludgy, and hopefully ext3 reservations won't defeat it.

Re: Slow swapon for big (12GB) swap

[Grzegorz Kulewski]

On Mon, 10 Apr 2006, Helge Hafting wrote:
> Grzegorz Kulewski wrote:
>
>>  Hi,
>>
>>  I am using big swap here (as a backing for potentially huge tmpfs). And I
>>  wonder why swapon on such big (like 12GB) swap takes about 7 minutes
>>  (continuous disk IO). Is this expected? Why it is like that? Can I do
>>  anything to speed it up? Or maybe remove it into the background with low
>>  priority or something like that?
>
> I don't know why it is slow.  But you can certainly do something like:
>
> nice swapon /dev/yourdisk &

I wouldn't be so sure about this because it is possible that the real work 
is done by some kernel thread not the process...


> Then it will happen in the background and with low priority.  Of course,
> you can't start filling your tmpfs until this completes.

That is no problem.


> I don't think tmpfs+swap was made with this sort of use in mind,
> so you may want to test the performance when you fill up such a
> tmpfs, and compare to the performance of /tmp on a 12GB
> ordinary filesystem.  It seems to me that the advantage of /tmp on
> tmpfs is lost completely if most of it has to be written to disk anyway.
> (Ordinary filesystems are cached too, the "tmpfs advantage" is that
> truly temporary (but possibly long-lived) files are never written
> to disk _if_ you have enough memory.  /tmp on a plain filesystem
> is just as fast due to caching, but may delay other use of the
> disk as the ordinary filesystem writes stuff out so it will be
> saved for the future.)

Well - I use it for /var/tmp for compiling packages in Gentoo. Most 
compiles use < 1MB of it and it is *much* faster that way because 
immendiate data never touch disk. And the disk stays idle the whole time 
so can be put to sleep and should live longer.

There are some compiles that take < 600MB and they are still done in RAM. 
And there are < 4 packages in this system that need extreeme amounts of 
disk (like openoffice for example). They of course are not done in RAM but 
it looks like they are still rather fast (but I don't have any 
meansurements). And the disk is still idle or nearly idle for the most 
part of these 8hs. Time shows that 6,8h is spent in userspace and I 
think it is pretty good.

So, to sum up, I need it in RAM and for the most part of the time (like 
99,99% at least) it is not filled more than 60MB. I could probably make it 
smaller and change to disk when such large package is going to be compiled 
but this will require changing Gentoo scripts or doing things manually and 
I want to avoid that.


Thanks,

Grzegorz Kulewski

Re: Slow swapon for big (12GB) swap

[Grzegorz Kulewski]

On Mon, 10 Apr 2006, Andrew Morton wrote:
> Grzegorz Kulewski <kangur@polcom.net> wrote:
>>
>> I am using big swap here (as a backing for potentially huge tmpfs). And I
>>  wonder why swapon on such big (like 12GB) swap takes about 7 minutes
>>  (continuous disk IO).
>
> It's a bit quicker here:
>
> vmm:/usr/src/25# mkswap /dev/hda6
> Setting up swapspace version 1, size = 54031826 kB
> vmm:/usr/src/25# time swapon /dev/hda6
> swapon /dev/hda6  0.00s user 0.04s system 74% cpu 0.054 total
>
>
>> Is this expected?
>
> Nope.
>
>> Why it is like that?
>
> Are you using a swapfile or a swap partition?

Swap file.


> If it's a swapfile then perhaps the filesystem is being inefficient in its
> bmap() function.  Which filesystem is it?

Ext3.

The kernel is 2.6.16-rc3-git2-ck1. There is -ck patch in it but I strongly 
hope that swap prefetch is not *that* buggy to cause such things... I am 
considering testing vanilla. Con CC'd.

The system is Athlon XP 2000MHz with 1GB of RAM and Samsung 80GB IDE hard 
drive.

The second run of swapon is very fast (like 2 seconds). So it looks like 
it has cached something in RAM?...


Thanks,

Grzegorz Kulewski

Re: Slow swapon for big (12GB) swap

[Con Kolivas]

On Monday 10 April 2006 20:25, Grzegorz Kulewski wrote:
> The kernel is 2.6.16-rc3-git2-ck1. There is -ck patch in it but I strongly
> hope that swap prefetch is not *that* buggy to cause such things... I am
> considering testing vanilla. Con CC'd.

Turn it off by setting the swap_prefetch tunable to 0. That disables even 
storing swapped information. Swap prefetch does very little with that set to 
0, but ultimately you'd have to de-configure it to be sure I guess.

-- 
-ck

Re: Slow swapon for big (12GB) swap

[Grzegorz Kulewski]

On Mon, 10 Apr 2006, Andrew Morton wrote:
> Grzegorz Kulewski <kangur@polcom.net> wrote:
>> On Mon, 10 Apr 2006, Andrew Morton wrote:
>>> Grzegorz Kulewski <kangur@polcom.net> wrote:
>>>> I am using big swap here (as a backing for potentially huge tmpfs). And I
>>>>  wonder why swapon on such big (like 12GB) swap takes about 7 minutes
>>>>  (continuous disk IO).
>>>
>>> It's a bit quicker here:
>>>
>>> vmm:/usr/src/25# mkswap /dev/hda6
>>> Setting up swapspace version 1, size = 54031826 kB
>>> vmm:/usr/src/25# time swapon /dev/hda6
>>> swapon /dev/hda6  0.00s user 0.04s system 74% cpu 0.054 total
>>>
>>>
>>>> Is this expected?
>>>
>>> Nope.
>>>
>>>> Why it is like that?
>>>
>>> Are you using a swapfile or a swap partition?
>>
>> Swap file.
>>
>>
>>> If it's a swapfile then perhaps the filesystem is being inefficient in its
>>> bmap() function.  Which filesystem is it?
>>
>> Ext3.
>>
>
> OK.  A 12G file is around 3000 indirect blocks.  At 10 milliseconds each
> I'd expect it to take 30-odd seconds.  A 14G swapfile here takes 20 second
> for swapon.

That is what I would expect.


> Maybe your disks are slow, or the fs is already quite full+fragmented.

Well, the disk is slow but not that slow... For example dd creating this 
file goes at about 20-25MB/s and takes about 10 minutes IIRC, so it is not 
*that* bad...

The disk is not full or even close to:
/dev/hda4          /mnt/data        52552,1  27343,6  24140,4  54%  ext3

and file count is not that high too:
/dev/hda4            6836224   42333 6793891    1% /mnt/data

and it should not be fragmented in any way. It is rather new (3 months 
old) filesystem with several bigger files and small amount of smaller. 
Used mainly as a data store and backup. Very infrequently written to (new 
blocks/files).

I think I will test new kernel soon. Maybe with and without -ck. Maybe 
disk scheduler (probably CFQ) does something stupid... Or swap prefetch... 
Or anything...


Thanks,

Grzegorz Kulewski

Re: Slow swapon for big (12GB) swap

[Jan Engelhardt]

>
> Well - I use it for /var/tmp for compiling packages in Gentoo. Most compiles
> use < 1MB of it and it is *much* faster that way because immendiate data never
> touch disk. And the disk stays idle the whole time so can be put to sleep and
> should live longer.
>

Spinning the disk up and down more often than a continuously-running disk 
is also a issue.


Jan Engelhardt
-- 

Re: Slow swapon for big (12GB) swap

[linux-os (Dick Johnson)]

On Tue, 11 Apr 2006, Jan Engelhardt wrote:

>>
>> Well - I use it for /var/tmp for compiling packages in Gentoo. Most compiles
>> use < 1MB of it and it is *much* faster that way because immendiate data never
>> touch disk. And the disk stays idle the whole time so can be put to sleep and
>> should live longer.
>>
>
> Spinning the disk up and down more often than a continuously-running disk
> is also a issue.
>
>
> Jan Engelhardt
> --

Yes, a spinning disk gathers no moss! Once a disk is running, the
heads are flying and the shaft remains centered in its bearing
by hydrostatic forces from the film of oil that circulates around
the "Oilite" (sintered bronze) bearings. There is virtually no
wear. It's the startup and shutdown that takes its toll. That's
why there are so may crashed PCs after a power failure at a
large company. The "@*(%^@_*(" disk drive(s) won't start up,
having been running fine for the past five or more years!

Cheers,
Dick Johnson
Penguin : Linux version 2.6.15.4 on an i686 machine (5589.42 BogoMips).
Warning : 98.36% of all statistics are fiction, book release in April.
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.