Question about memory pressure freezes

Question about memory pressure freezes

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

I have often experienced nearly full system freezes for up to five
seconds at a time while memory is being flushed to disk.

I'm not sure if this is a general Linux problem or a Reiser4 problem, so
I thought I'd ask.

A sysrq-T during the freeze shows many processes trying to acquire a
memory page, and Reiser4 flushing atoms and doing sync things.

My working theory right now is that Reiser4 spends time flushing a lot
of data at once before returning.

If I am right about that, would it not make more sense to flush a few
pages, return to the kernel, flush a few more pages, return to the
kernel, etc, etc?  That way programs could get a bit of RAM and make
some progress.

Of course I could be completely off about what's going on.
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Question about memory pressure freezes

[Edward Shishkin]

Zan Lynx wrote:

>I have often experienced nearly full system freezes for up to five
>seconds at a time while memory is being flushed to disk.
>
>I'm not sure if this is a general Linux problem or a Reiser4 problem, so
>I thought I'd ask.
>
>  
>

I wouldn't separate the only one of them to blame ;)

>A sysrq-T during the freeze shows many processes trying to acquire a
>memory page, and Reiser4 flushing atoms and doing sync things.
>  
>

yup, and all processes (including interactive ones) wait for 
reiser4_writepage
completion.. This takes place for mapped writes, which can not balance dirty
pages like usual writes..

>My working theory right now is that Reiser4 spends time flushing a lot
>of data at once before returning.
>  
>

right,
there was a lightweight reaction on memory pressure notification (emergency
flush) a long time ago, but Hans couldn't live with such "swapping" 
mechanism.
Also the emergency flush itself had some problems.

>If I am right about that, would it not make more sense to flush a few
>pages, return to the kernel, flush a few more pages, return to the
>kernel, etc, etc?  That way programs could get a bit of RAM and make
>some progress.
>  
>

I have spent 1-2 days of moving in this direction: no progress.
I wouldn't say that this is ultimately wrong way: just requires
more time to investigate..

>Of course I could be completely off about what's going on.
>  
>

Re: Question about memory pressure freezes

[Xu CanHao]

Hello!

    Since the default atom_max_size is RAM/4, that means 256MB of
maximum atom in your computer, and your HDD does 30MB/s which means
you'll have 8s-9s "flush/system freeze" time.

    IIRC, edward said the atom_max_size mount option is useless, so
I'd suggest you modify your reiser4/init_super.c and change:
       sbinfo->tmgr.atom_max_size = totalram_pages / 4;
to
       sbinfo->tmgr.atom_max_size = totalram_pages / 16;
which means your atom would have a maximum size of 64MB and a maximum
freeze time of only 2 seconds.

Thanks!


2007/7/28, Zan Lynx <zlynx@acm.org>:
> Edward answered my question mostly.
>
> But, I have 1 GB RAM and a Hitachi 7,200 RPM laptop drive.  It does
> about 30 MB/s.
>
> On Fri, 2007-07-27 at 09:40 +0800, Xu CanHao wrote:
> > Hello!
> >
> >    How much is your RAM?
> >    What is the result of your HDD's hdparm -t?
> >
> > Thanks!
> >
> > 2007/7/27, Zan Lynx <zlynx@acm.org>:
> > > I have often experienced nearly full system freezes for up to five
> > > seconds at a time while memory is being flushed to disk.
> > >
> > > I'm not sure if this is a general Linux problem or a Reiser4 problem, so
> > > I thought I'd ask.
> > >
> > > A sysrq-T during the freeze shows many processes trying to acquire a
> > > memory page, and Reiser4 flushing atoms and doing sync things.
> > >
> > > My working theory right now is that Reiser4 spends time flushing a lot
> > > of data at once before returning.
> > >
> > > If I am right about that, would it not make more sense to flush a few
> > > pages, return to the kernel, flush a few more pages, return to the
> > > kernel, etc, etc?  That way programs could get a bit of RAM and make
> > > some progress.
> > >
> > > Of course I could be completely off about what's going on.
> > > --
> > > Zan Lynx <zlynx@acm.org>
> > >
> > >
> --
> Zan Lynx <zlynx@acm.org>
>
>

Re: Question about memory pressure freezes

[Edward Shishkin]

Xu CanHao wrote:

>Hello!
>
>    Since the default atom_max_size is RAM/4, that means 256MB of
>maximum atom in your computer, and your HDD does 30MB/s which means
>you'll have 8s-9s "flush/system freeze" time.
>
>    IIRC, edward said the atom_max_size mount option is useless,
>

It is  useless for remount, in other cases it works fine
(for root file system it makes sense to edit fstab).
So you can pass the option "tmgr.atom_max_size=N",
where N  is atom size in blocks

> so
>I'd suggest you modify your reiser4/init_super.c and change:
>       sbinfo->tmgr.atom_max_size = totalram_pages / 4;
>to
>       sbinfo->tmgr.atom_max_size = totalram_pages / 16;
>which means your atom would have a maximum size of 64MB and a maximum
>freeze time of only 2 seconds.
>  
>

Such nice linear dependency disappears with some workload,
so freeze time is not comforting also for small atoms..

>Thanks!
>
>
>2007/7/28, Zan Lynx <zlynx@acm.org>:
>  
>
>>Edward answered my question mostly.
>>
>>But, I have 1 GB RAM and a Hitachi 7,200 RPM laptop drive.  It does
>>about 30 MB/s.
>>
>>On Fri, 2007-07-27 at 09:40 +0800, Xu CanHao wrote:
>>    
>>
>>>Hello!
>>>
>>>   How much is your RAM?
>>>   What is the result of your HDD's hdparm -t?
>>>
>>>Thanks!
>>>
>>>2007/7/27, Zan Lynx <zlynx@acm.org>:
>>>      
>>>
>>>>I have often experienced nearly full system freezes for up to five
>>>>seconds at a time while memory is being flushed to disk.
>>>>
>>>>I'm not sure if this is a general Linux problem or a Reiser4 problem, so
>>>>I thought I'd ask.
>>>>
>>>>A sysrq-T during the freeze shows many processes trying to acquire a
>>>>memory page, and Reiser4 flushing atoms and doing sync things.
>>>>
>>>>My working theory right now is that Reiser4 spends time flushing a lot
>>>>of data at once before returning.
>>>>
>>>>If I am right about that, would it not make more sense to flush a few
>>>>pages, return to the kernel, flush a few more pages, return to the
>>>>kernel, etc, etc?  That way programs could get a bit of RAM and make
>>>>some progress.
>>>>
>>>>Of course I could be completely off about what's going on.
>>>>--
>>>>Zan Lynx <zlynx@acm.org>
>>>>
>>>>
>>>>        
>>>>
>>--
>>Zan Lynx <zlynx@acm.org>
>>
>>
>>    
>>
>-
>To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>  
>

Re: Question about memory pressure freezes

[Xu CanHao]

2007/7/29, Edward Shishkin <edward@namesys.com>:
> Xu CanHao wrote:
>
> >Hello!
> >
> >    Since the default atom_max_size is RAM/4, that means 256MB of
> >maximum atom in your computer, and your HDD does 30MB/s which means
> >you'll have 8s-9s "flush/system freeze" time.
> >
> >    IIRC, edward said the atom_max_size mount option is useless,
> >
>
> It is  useless for remount, in other cases it works fine
> (for root file system it makes sense to edit fstab).
> So you can pass the option "tmgr.atom_max_size=N",
> where N  is atom size in blocks

IIRC months ago I asked you about something that when I pass the
option "tmgr.atom_max_size=N" to /etc/fstab(root partition) and
/proc/mounts still shows the original RAM/4 value, you said it's
mysteries and a long-term object.

But the option worked with newly mounted partition such as "#mount
.... -o tmgr.atom_max_size=N"

>
> > so
> >I'd suggest you modify your reiser4/init_super.c and change:
> >       sbinfo->tmgr.atom_max_size = totalram_pages / 4;
> >to
> >       sbinfo->tmgr.atom_max_size = totalram_pages / 16;
> >which means your atom would have a maximum size of 64MB and a maximum
> >freeze time of only 2 seconds.
> >
> >
>
> Such nice linear dependency disappears with some workload,
> so freeze time is not comforting also for small atoms..

Then maybe a value of RAM/128 or 256 or 512... small enough to make
the freeze time short enough...  ;)

>
> >Thanks!
> >
> >
> >2007/7/28, Zan Lynx <zlynx@acm.org>:
> >
> >
> >>Edward answered my question mostly.
> >>
> >>But, I have 1 GB RAM and a Hitachi 7,200 RPM laptop drive.  It does
> >>about 30 MB/s.
> >>
> >>On Fri, 2007-07-27 at 09:40 +0800, Xu CanHao wrote:
> >>
> >>
> >>>Hello!
> >>>
> >>>   How much is your RAM?
> >>>   What is the result of your HDD's hdparm -t?
> >>>
> >>>Thanks!
> >>>
> >>>2007/7/27, Zan Lynx <zlynx@acm.org>:
> >>>
> >>>
> >>>>I have often experienced nearly full system freezes for up to five
> >>>>seconds at a time while memory is being flushed to disk.
> >>>>
> >>>>I'm not sure if this is a general Linux problem or a Reiser4 problem, so
> >>>>I thought I'd ask.
> >>>>
> >>>>A sysrq-T during the freeze shows many processes trying to acquire a
> >>>>memory page, and Reiser4 flushing atoms and doing sync things.
> >>>>
> >>>>My working theory right now is that Reiser4 spends time flushing a lot
> >>>>of data at once before returning.
> >>>>
> >>>>If I am right about that, would it not make more sense to flush a few
> >>>>pages, return to the kernel, flush a few more pages, return to the
> >>>>kernel, etc, etc?  That way programs could get a bit of RAM and make
> >>>>some progress.
> >>>>
> >>>>Of course I could be completely off about what's going on.
> >>>>--
> >>>>Zan Lynx <zlynx@acm.org>
> >>>>
> >>>>
> >>>>
> >>>>
> >>--
> >>Zan Lynx <zlynx@acm.org>
> >>
> >>
> >>
> >>
> >-
> >To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> >
> >
>
>