umount and sync delays

umount and sync delays

[Gorazd Golob]

hi!

I have strangely long delay (still waiting at the time) when unmounting or 
syncing partition with reiser4. At the same time writes are occuring to 
disk subsystem - when checking vmstat and of course checking disk activity 
light ;).

Why does it take so long and it will ever finish ?:)
I don't have debug enabled and I'm using 2.6.11-mm1 kernel. Partitions 
were formated with reiser4progs 1.0.4.

gorazd

Re: umount and sync delays

[Hans Reiser]

Gorazd Golob wrote:

> hi!
>
> I have strangely long delay (still waiting at the time) when
> unmounting or syncing partition with reiser4. At the same time writes
> are occuring to disk subsystem - when checking vmstat and of course
> checking disk activity light ;).
>
> Why does it take so long and it will ever finish ?:)
> I don't have debug enabled and I'm using 2.6.11-mm1 kernel. Partitions
> were formated with reiser4progs 1.0.4.
>
> gorazd
>
>
Probably you just have a lot of dirty buffers in RAM and they need to
flush to disk?

Re: umount and sync delays

[Gorazd Golob]

Hi!

>>
> Probably you just have a lot of dirty buffers in RAM and they need to
> flush to disk?
>
Hmm.. I'll try explain .. Before I did sync or unmount, application, that 
previosly wrote to reiser4 partition was closed. In moment of running sync 
or umount OS had about 1.7gb of data in cache memory. Sync was finished 
in about 45 minutes after. I'm not sure why does it take so long on u320s 
scsi. The problem can be in files which are written to this partition 
- a lot (>1M) of really small files. . Do we have to use some mount arguments?
It's not really practical if server goes out of power and ups will 
last only 5 minutes - not enought time to safe shutdown of system..

Thanks, Gorazd

Re: umount and sync delays

[Hans Reiser]

Gorazd Golob wrote:

> Hi!
>
>>>
>> Probably you just have a lot of dirty buffers in RAM and they need to
>> flush to disk?
>>
> Hmm.. I'll try explain .. Before I did sync or unmount, application,
> that previosly wrote to reiser4 partition was closed. In moment of
> running sync or umount OS had about 1.7gb of data in cache memory.
> Sync was finished in about 45 minutes after. I'm not sure why does it
> take so long on u320s scsi. The problem can be in files which are
> written to this partition - a lot (>1M) of really small files. . Do we
> have to use some mount arguments?

I have no idea, and so I must guess.  I guess that if you try the same
application on a different computer it will work reasonably fast and
that the problem is hardware doing extensive retries. 

Or, you are doing completely random IO with tiny little bits of dirty
data in a large file, and so each 4k is a seek.  I doubt this, but in
theory....

Or you have something else on that machine dirtying lots of memory after
the application closes, and perhaps there is a flaw in our method for
forcing atoms to disk after they get old that we should look into.

Finally, it could be none of the above and something is wrong in our
sync algorithms, and somebody should log onto your machine and look at it.

Probably all guesses are wrong, but I have to ask them as a start.

Zam can probably login and look at things for you.  Zam, this guy is a
friend of mine, could you do that?

> It's not really practical if server goes out of power and ups will
> last only 5 minutes - not enought time to safe shutdown of system..
>
> Thanks, Gorazd
>
>

Re: umount and sync delays

[Gorazd Golob]

>
> I have no idea, and so I must guess.  I guess that if you try the same
> application on a different computer it will work reasonably fast and
> that the problem is hardware doing extensive retries.
We have tests on ext3 and it wasn't like fast, but few minutes, not 30 
minutes.. 
>
> Or, you are doing completely random IO with tiny little bits of dirty
> data in a large file, and so each 4k is a seek.  I doubt this, but in
> theory....
No.. there is really "random" data in this files, but files are small 
(most of them less than 4 kb).

> Or you have something else on that machine dirtying lots of memory after
> the application closes, and perhaps there is a flaw in our method for
> forcing atoms to disk after they get old that we should look into.
There were no other applications on that system and none of user land 
deamons were using that partition.
>
> Finally, it could be none of the above and something is wrong in our
> sync algorithms, and somebody should log onto your machine and look at it.
Thanks.
>
> Probably all guesses are wrong, but I have to ask them as a start.
Hope it helped..

tnx, gorazd

Re: umount and sync delays

[Alex Zarochentsev]

Hello,

On Mon, Mar 14, 2005 at 08:59:00PM +0100, Gorazd Golob wrote:
> >
> >I have no idea, and so I must guess.  I guess that if you try the same
> >application on a different computer it will work reasonably fast and
> >that the problem is hardware doing extensive retries.
> We have tests on ext3 and it wasn't like fast, but few minutes, not 30 
> minutes.. 
> >
> >Or, you are doing completely random IO with tiny little bits of dirty
> >data in a large file, and so each 4k is a seek.  I doubt this, but in
> >theory....
> No.. there is really "random" data in this files, but files are small 
> (most of them less than 4 kb).

can you repeat the test with earlier kernel, like 
2.6.9 + the patch from ftp.namesys.com ?

just to be sure that the bug wasn't introduced recently.

> >Or you have something else on that machine dirtying lots of memory after
> >the application closes, and perhaps there is a flaw in our method for
> >forcing atoms to disk after they get old that we should look into.
> There were no other applications on that system and none of user land 
> deamons were using that partition.
> >
> >Finally, it could be none of the above and something is wrong in our
> >sync algorithms, and somebody should log onto your machine and look at it.
> Thanks.

It is better to find how to reproduce the bug in our test environment 
(kGDB :-).  

> >
> >Probably all guesses are wrong, but I have to ask them as a start.
> Hope it helped..

if you have managed to create tons of independed atoms, tons of independed
commits may take a lot of time :(

> tnx, gorazd

-- 
Alex.