quicker mount

quicker mount

[Andrea Gelmini]

Hi all,
	mounting 1.9TB partitions (latest stable kernel), takes 3m33s.
	What can I do to make it quicker?

Thanks for your time,
gelma

Re: quicker mount

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrea Gelmini wrote:
> Hi all,
> 	mounting 1.9TB partitions (latest stable kernel), takes 3m33s.
> 	What can I do to make it quicker?

It takes so long to mount since ReiserFS reads all the bitmaps before
mounting. The larger the file system, the longer it takes.

I have patches that dynamically load the bitmaps, which makes the mount
nearly instantaneous, but I haven't finished debugging them.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFEUNhdLPWxlyuTD7IRAq7nAJ9gwm52meieN9sb2aQ+q4BczIZTEQCfXPGp
VIe3X23Uy20b33CfjC+AJh8=
=Di4u
-----END PGP SIGNATURE-----

Re: quicker mount

[Michael Weissenbacher]

Hi,
> It takes so long to mount since ReiserFS reads all the bitmaps before
> mounting. The larger the file system, the longer it takes.
I wonder why it has to read all of them. Are they stored in RAM after 
reading? Otherwise I cannot see why it should be done at all.

with kind regards,
-- 
/============================================================\
|       Michael Weissenbacher <webmaster@dermichi.com>       |
|                  http://www.dermichi.com/                  |
| "The trouble with computers is that they do what you tell  |
|           them, not what you want." -- D. Cohen            |
\============================================================/

Re: quicker mount

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Weissenbacher wrote:
> Hi,
>> It takes so long to mount since ReiserFS reads all the bitmaps before
>> mounting. The larger the file system, the longer it takes.
> I wonder why it has to read all of them. Are they stored in RAM after
> reading? Otherwise I cannot see why it should be done at all.

Yes, they're kept in RAM. You can see the debate on this in the list
archives, but I asked the same questions.

There is no reason for the bitmaps to be retained in memory, as they're
more rarely used than other metadata which simply relys on the kernel to
cache intelligently.

The patches are written, they just need some debugging. Unfortunately,
they're kind of on the back burner right now due to other work constraints.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFEUOY8LPWxlyuTD7IRAguzAKCiudFGY1tFXOiSJ1743lrrb5kBJgCfcVG/
a3YBOD1EVcrEGsDFgkWRkKs=
=9lHG
-----END PGP SIGNATURE-----

Re: quicker mount

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeff Mahoney wrote:
> Michael Weissenbacher wrote:
>>> Hi,
>>>> It takes so long to mount since ReiserFS reads all the bitmaps before
>>>> mounting. The larger the file system, the longer it takes.
>>> I wonder why it has to read all of them. Are they stored in RAM after
>>> reading? Otherwise I cannot see why it should be done at all.
> 
> Yes, they're kept in RAM. You can see the debate on this in the list
> archives, but I asked the same questions.
> 
> There is no reason for the bitmaps to be retained in memory, as they're
> more rarely used than other metadata which simply relys on the kernel to
> cache intelligently.
> 
> The patches are written, they just need some debugging. Unfortunately,
> they're kind of on the back burner right now due to other work constraints.

To elaborate, there are places where the bitmap allocation code isn't
allowed to sleep. The existing code doesn't sleep, since all the bitmaps
are cached in memory. Allowing the bitmap blocks to be dropped from the
cache means that sometimes the blocks will need to be re-read, causing a
schedule to occur while the data is read from disk.

It is the schedule itself that causes the problem, not the bitmap block
getting re-read from disk. By backing out the patch that actually loads
the bitmaps dynamically and adding an msleep(30) in the path where the
bitmap blocks are used, I can reproduce the same failure as if the
blocks were loaded dynamically. In both cases it can take anywhere from
a few hours to a few days to trigger on a heavily loaded machine. My
test case has been a rather unrealistic 50-process copy-delete loop from
/usr/include to a test file system sized to create bitmap contention on
a 4 CPU machine with the RAM limited to 128 MB.

At any rate, even with the relative infrequency of the failures, it's
still a regression and I wouldn't consider submitting to mainline
(again) until they've been fixed.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFEVnHqLPWxlyuTD7IRAqUUAJ4221Icf42yiDpg8OrZu5wgGYS7xgCcCCao
wnPpE66IkL5UEGv82Ylv1fw=
=bFsD
-----END PGP SIGNATURE-----

Re: quicker mount

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeff Mahoney wrote:
> Jeff Mahoney wrote:
>>> Michael Weissenbacher wrote:
>>>>> Hi,
>>>>>> It takes so long to mount since ReiserFS reads all the bitmaps before
>>>>>> mounting. The larger the file system, the longer it takes.
>>>>> I wonder why it has to read all of them. Are they stored in RAM after
>>>>> reading? Otherwise I cannot see why it should be done at all.
>>> Yes, they're kept in RAM. You can see the debate on this in the list
>>> archives, but I asked the same questions.
>>>
>>> There is no reason for the bitmaps to be retained in memory, as they're
>>> more rarely used than other metadata which simply relys on the kernel to
>>> cache intelligently.
>>>
>>> The patches are written, they just need some debugging. Unfortunately,
>>> they're kind of on the back burner right now due to other work constraints.
> 
> To elaborate, there are places where the bitmap allocation code isn't
> allowed to sleep. The existing code doesn't sleep, since all the bitmaps
> are cached in memory. Allowing the bitmap blocks to be dropped from the
> cache means that sometimes the blocks will need to be re-read, causing a
> schedule to occur while the data is read from disk.
> 
> It is the schedule itself that causes the problem, not the bitmap block
> getting re-read from disk. By backing out the patch that actually loads
> the bitmaps dynamically and adding an msleep(30) in the path where the
> bitmap blocks are used, I can reproduce the same failure as if the
> blocks were loaded dynamically. In both cases it can take anywhere from
> a few hours to a few days to trigger on a heavily loaded machine. My
> test case has been a rather unrealistic 50-process copy-delete loop from
> /usr/include to a test file system sized to create bitmap contention on
> a 4 CPU machine with the RAM limited to 128 MB.
> 
> At any rate, even with the relative infrequency of the failures, it's
> still a regression and I wouldn't consider submitting to mainline
> (again) until they've been fixed.

I'm embarrassed to say that after staring at the code all afternoon, my
analysis was completely wrong. I enabled CONFIG_REISERFS_CHECK to try to
catch where it was failing earlier, and the code panicked immediately
when trying to write to the file system. The error was the same as I was
seeing in tests, just caught earlier.

It turns out that the third patch in the series was flawed and didn't
treat the v3.[56] bitmap 0 specially. Every other bitmap block on the
file system is (blocksize * 8 * bitmap number), except bitmap 0 which is
the block after the superblock. I was mistakenly loading block 0 as the
bitmap block, which is obviously wrong.

This explains why I couldn't seem to find any problems related to
scheduling as well as why the panics always involved block 8211. I'll be
running some overnight stress tests as well as performing a full audit
of the code which was essentially moved around to ensure more of these
stupid bugs didn't sneak in.

I'll post the updated patches tomorrow, but they still have the caveat
that the error handling infrastructure just isn't there yet. I guess
I'll resurrect those next. :)

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFEVtZdLPWxlyuTD7IRAhxrAJwIma2pJQ3WkplSftEws8SvlJEfHQCfVh57
ntK3GxhfHRRdeFQwKjTyK04=
=DPQv
-----END PGP SIGNATURE-----

Re: quicker mount

[Tom Vier]

What about reiserfs4? I just tried a 250gig raid1 and it r4 takes even
longer than 3. Is it also preloading all bitmaps? No other fs that i've
tried takes so long to mount.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: quicker mount

[Vladimir V. Saveliev]

Hello

On Tue, 2006-05-23 at 10:43 -0400, Tom Vier wrote:
> What about reiserfs4? I just tried a 250gig raid1 and it r4 takes even
> longer than 3. Is it also preloading all bitmaps? No other fs that i've
> tried takes so long to mount.
> 
reiser4 has mount option for that.
please try mount -o dont_load_bitmap