some notes on a memory leak

some notes on a memory leak

[Gergely Gábor]

[-- Attachment #1.1: Type: text/plain, Size: 2208 bytes --]

Hello Ryusuke, and other nilfs developers, and users!

I have left my computer working, testing nilfs. I have found that after a log period of time, most of the memory in the computer vas allocated as cache (according to vmstat) and even aftel killing all processes that held files open, the amount (~430 megabytes) did not decrease significantly. I have unmounted the filesystem, and most cache was freed up, and after remounting nilfs, the system was fast as lightning again (the module was not needed to be removed to reclaim the memory). This leads to the assumption, that some data is left cached, even when not in use. To reproduce this behaviour, I simply started "vmstat 1", and rtorrent, started a download, and watched the cache consumption climb. after exiting rtorrent, most of the cache is not reclaimed, not even after a long period of time. (i understand, that leaving blocks in the cache is preferable for a log structured fs, as that reduces seeks necessary for reading, but possibly some aging mechanism, and/or a maximum cache size option would be a good idea to implement in the future, if not yet done)

Apart from this minor issue, the fs is working fine. i feel that (but this is very subjective, only based on my perception) the current rate of this leakage is far smaller that the rate i experienced with the patched version of testing-8. (than the system went unusable after a few minutes, as all mem was consumed, not it took more than 4 hours, as i wrote a letter with fairly much free mem before leaving home with the machine running.)

Sorry for flooding the mail list with my self-contradictory status reports.

Please tell me, where can i read about the architecture of nilfs2 in greater detail, other than the code? (i'm willing to understand the code as well, but i'd need some guidance, where to start. I'd be the most happy if i could provide some more useful help rather than constant "complaining") has many things changed since nilfs1? (i have found papers about that on the project homepage)

Yours sincerely: Gergely Gábor
-- 
Gergely Gábor <elentirmo.gilgalad-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

* random fortune:
Rome wasn't burnt in a day.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: some notes on a memory leak

[Ryusuke Konishi]

Hi Gábor,

On Wed, 2008-02-13 at 15:40 +0100, Gergely Gábor wrote:
> This release works flawlessly for me, for over 4 hours now. No memory
> leak, no deadlock occured, as I tested the previously malfunctioning
> scenarios.

Good news!
We might finally see daylight for the stable release.

On Thu, 2008-02-14 at 01:02 +0100, Gergely Gábor wrote:
> I have left my computer working, testing nilfs. I have found that 
> after a log period of time, most of the memory in the computer vas 
> allocated as cache (according to vmstat) and even aftel killing all
>  processes that held files open, the amount (~430 megabytes) did not
>  decrease significantly. I have unmounted the filesystem, and most
>  cache was freed up, ...
>  This leads to the assumption, that some data is left cached, even 
> when not in use. To reproduce this behaviour, I simply started 
> "vmstat 1", and rtorrent, started a download, and watched the cache
>  consumption climb. after exiting rtorrent, most of the cache is not
> reclaimed, not even after a long period of time. 

Yeah, this is the peculiar behaviour of Linux memory management.
Linux try to devote free memory to the page cache wherever possible.

> (i understand, that leaving blocks in the cache is preferable for
> a log structured fs, as that reduces seeks necessary for reading,
> but possibly some aging mechanism, and/or a maximum cache size
> option would be a good idea to implement in the future, if not yet done)

As I commented above, the bahaviour that leaves blocks in the cache,
is the nature of Linux, and basically its control is supposed to
be the matter of the Linux mm and applications.
For example, applications can invalidate the cache through
posix_fadvice() or madvise() system call, and users can tune
the cache behaviour through /sbin/sysctl vm.* or /proc/sys/vm/*.

Of course, we are taking full advantage of the nature for NILFS to
compensate the drawback that logfs incurs performance penalty 
for reads.  And I believe that this cache behaviour is harmless 
because most of cached pages are reclaimable any time if not modified. 

However, a memory leak must be distinguished because it blows up
this scheme. I'm nervous about it if suspicious.
So, now I'm relieved to hear your report.

> Sorry for flooding the mail list with my self-contradictory status reports.

Don't mind. Your reports are very helpful.

> Please tell me, where can i read about the architecture of nilfs2 
> in greater detail, other than the code? (i'm willing to understand 
> the code as well, but i'd need some guidance, where to start. 
> I'd be the most happy if i could provide some more useful help 
> rather than constant "complaining") has many things changed since
>  nilfs1? (i have found papers about that on the project homepage)

One of my colleage is writing a paper of nilfs2, and I believe it
would satisify these requests.

Sincerely,
-- 
Ryusuke Konishi
NILFS team NTT
http://www.nilfs.org/

Re: some notes on a memory leak

[Ryusuke Konishi]

From: Gergely Gábor <elentirmo.gilgalad-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Subject: [NILFS users] some notes on a memory leak
Date: Thu, 14 Feb 2008 01:02:58 +0100

> Hello Ryusuke, and other nilfs developers, and users!
> 
> I have left my computer working, testing nilfs. I have found that after
> a log period of time, most of the memory in the computer vas allocated as
> cache (according to vmstat) and even aftel killing all processes that held
> files open, the amount (~430 megabytes) did not decrease significantly.
> I have unmounted the filesystem, and most cache was freed up, and after
> remounting nilfs, the system was fast as lightning again (the module was not
> needed to be removed to reclaim the memory).

I'd like to append a note.

NILFS has two different things about the cache usage than ordinary
filesystems:

(1) Since the cleaner clowls over a partition, it adds new pages into a cache
    of the block device one after another during it operates on the background.

(2) Some blocks, for example the blocks to store header information of each
    segment, are allocated incrementally into the cache of the block device. 
    So, bursty writes or frequent sync writes may bloat it.
    Conversely, the turnover of buffers for files or meta data should not be 
    higher than usual fs because a same buffer serves for a same file offset
    even when its physical disk address changes.

I made a patch to minimize the bloat of bdev cache due to the second
difference.
If you are bothered with that, please test the following patch.
From my local tests, it didn't make no remarkable difference.

Best regards,
-- 
Ryusuke Konishi
NILFS team NTT
http://www.nilfs.org/

Index: fs/kern_feature.h
===================================================================
--- fs.orig/kern_feature.h	2008-02-14 15:28:15.000000000 +0900
+++ fs/kern_feature.h	2008-02-14 15:35:53.000000000 +0900
@@ -187,6 +187,13 @@
 	(LINUX_VERSION_CODE < KERNEL_VERSION(2,6,21))
 #endif
 /*
+ * invalidate_mapping_pages() got exported in linux-2.6.21.
+ */
+#ifndef HAVE_EXPORTED_INVALIDATE_MAPPING_PAGES
+# define HAVE_EXPORTED_INVALIDATE_MAPPING_PAGES \
+	(LINUX_VERSION_CODE > KERNEL_VERSION(2,6,20) || NILFS_BUILT_INTERNAL)
+#endif
+/*
  * clear_page_dirty() and test_clear_page_dirty() were removed
  * in linux-2.6.20
  */
Index: fs/segment.c
===================================================================
--- fs.orig/segment.c	2008-02-14 15:36:57.000000000 +0900
+++ fs/segment.c	2008-02-14 16:17:43.000000000 +0900
@@ -1615,11 +1615,19 @@
 
 static int nilfs_segctor_terminate_segment(struct nilfs_sc_info *sci,
 					   struct nilfs_segment_buffer *segbuf,
-					   struct inode *sufile)
+					   struct the_nilfs *nilfs)
 {
 	struct nilfs_segment_entry *ent = segbuf->sb_segent;
+	struct inode *sufile = nilfs->ns_sufile;
 	int err;
+#if HAVE_EXPORTED_INVALIDATE_MAPPING_PAGES
+	struct address_space *mapping = nilfs->ns_bdev->bd_inode->i_mapping;
+	int nshiftbits = PAGE_SHIFT - nilfs->ns_blocksize_bits;
+	pgoff_t start = segbuf->sb_fseg_start >> nshiftbits;
+	pgoff_t end = segbuf->sb_fseg_end >> nshiftbits;
 
+	invalidate_mapping_pages(mapping, start, end);
+#endif
 	err = nilfs_open_segment_entry(ent, sufile);
 	if (unlikely(err))
 		return err;
@@ -1669,7 +1677,7 @@
 		return err;
 
 	if (segbuf->sb_rest_blocks < NILFS_PSEG_MIN_BLOCKS) {
-		err = nilfs_segctor_terminate_segment(sci, segbuf, sufile);
+		err = nilfs_segctor_terminate_segment(sci, segbuf, nilfs);
 		if (unlikely(err))
 			return err;
 

Re: some notes on a memory leak

[Ryusuke Konishi]

On Fri, 2008-02-15 at 15:07 +0900, Ryusuke Konishi wrote:
> I made a patch to minimize the bloat of bdev cache due to the second
> difference.
> If you are bothered with that, please test the following patch.
> From my local tests, it didn't make no remarkable difference.

I dropped an important notice.
The patch is only valid against the kernel 2.6.21 and later.

Thanks,
-- 
Ryusuke Konishi
NILFS team NTT
http://www.nilfs.org/

Re: some notes on a memory leak

[Gergely Gábor]

[-- Attachment #1.1: Type: text/plain, Size: 2195 bytes --]

Greetings!
> 
> I'd like to append a note.
> 
> NILFS has two different things about the cache usage than ordinary
> filesystems:
> 
> (1) Since the cleaner clowls over a partition, it adds new pages into a cache
>     of the block device one after another during it operates on the background.
> 
> (2) Some blocks, for example the blocks to store header information of each
>     segment, are allocated incrementally into the cache of the block device. 
>     So, bursty writes or frequent sync writes may bloat it.
>     Conversely, the turnover of buffers for files or meta data should not be 
>     higher than usual fs because a same buffer serves for a same file offset
>     even when its physical disk address changes.
> 
generally nilfs2 performs well. rtorrent triggers some error.

> I made a patch to minimize the bloat of bdev cache due to the second
> difference.
> If you are bothered with that, please test the following patch.
> From my local tests, it didn't make no remarkable difference.
I tried the patch, i see no basic difference. the fs generally works fine, but rtorrent can make it consume all memory, and not free it after exit. just as if the cache entries didn't age (well i don't know how to describe the situation better with my naive words). I have tried setting the following, with no luck:
 vm.vfs_cache_pressure = 200
and even 
 vm.drop_caches = 3 

according to docs, it should have had some effect, but neither had any (in case of rtorrent's garbage)

the "pages" cached by rtorrent can't be flushed out of cache even when rtorrent has quit. This may as well be he fault of rtorrent, but I can remember an ext3 bug that was found by an rtorrent malfunction, and still, it is not right. (other programs generally seem to release cache after some time after having been closed). If i start a hash check on a near complete dvd image in rtorrent (0.7.9 version) i can waste my ram in less than 2 minutes!

umount helps. but now i can do my works on my notebook aswell.

Best Regards:
-- 
Gergely Gábor <elentirmo.gilgalad-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

* random fortune:
Above all things, reverence yourself.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users