Re: UBIFS assert failed in ubifs_set_page_dirty at 1421

Re: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Artem Bityutskiy]

Hi,

first of all, what is your architecture? ARM? And how easily can you
reproduce this? And can you try a kernel newer than 3.10?

And for fs-devel and mm people, here is the link to the original report:
http://lists.infradead.org/pipermail/linux-mtd/2014-October/055930.html, 

On Mon, 2014-10-20 at 12:01 +0000, Caizhiyong wrote:
> Here is part of the log, linux version 3.10:
>    cache 16240kB is below limit 16384kB for oom_score_adj 529
>    Free memory is -1820kB above reserved
> lowmemorykiller: Killing '.networkupgrade' (6924), adj 705,
>    to free 20968kB on behalf of 'kswapd0' (543) because
>    cache 16240kB is below limit 16384kB for oom_score_adj 529
>    Free memory is -2192kB above reserved

OK, no memory and OOM starts. So your system is in trouble anyway :-)

> UBIFS assert failed in ubifs_set_page_dirty at 1421 (pid 543)

UBIFS complain here that someone marks a page as dirty "directly", not
through one of the UBIFS functions. And that someone is the page reclaim
path.

Now, I do not really know what is going on here, so I am CCing a couple
of mailing lists, may be someone will help.

Here is what I see is going on.

1. UBIFS wants to make sure that no one marks UBIFS-backed pages (and
actually inodes too) as dirty directly. UBIFS wants everyone to ask
UBIFS to mark a page as dirty.

2. This is because for every dirty page, UBIFS needs to reserve certain
amount of space on the flash media, because all writes are out-of-place,
even when you are changing an existing file.

3. There are exactly 2 places where UBIFS-backed pages may be marked as
dirty:

  a) ubifs_write_end() [->wirte_end] - the file write path
  b) ubifs_page_mkwrite() [->page_mkwirte] - the file mmap() path

4. If anything calls 'ubifs_set_page_dirty()' directly (not through
write_end()/mkwrite()), and the page was not dirty, UBIFS will complain
with the assertion that you see.

> CPU: 3 PID: 543 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #1
> [<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>] (show_stack+0x20/0x24)
> [<80019f44>] (show_stack+0x20/0x24) from [<80af2ef8>] (dump_stack+0x24/0x2c)
> [<80af2ef8>] (dump_stack+0x24/0x2c) from [<80297234>] (ubifs_set_page_dirty+0x54/0x5c)
> [<80297234>] (ubifs_set_page_dirty+0x54/0x5c) from [<800cea60>] (set_page_dirty+0x50/0x78)
> [<800cea60>] (set_page_dirty+0x50/0x78) from [<800f4be4>] (try_to_unmap_one+0x1f8/0x3d0)
> [<800f4be4>] (try_to_unmap_one+0x1f8/0x3d0) from [<800f4f44>] (try_to_unmap_file+0x9c/0x740)
> [<800f4f44>] (try_to_unmap_file+0x9c/0x740) from [<800f5678>] (try_to_unmap+0x40/0x78)
> [<800f5678>] (try_to_unmap+0x40/0x78) from [<800d6a04>] (shrink_page_list+0x23c/0x884)
> [<800d6a04>] (shrink_page_list+0x23c/0x884) from [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8)
> [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8) from [<800d7c20>] (shrink_lruvec+0x3ac/0x524)
> [<800d7c20>] (shrink_lruvec+0x3ac/0x524) from [<800d8970>] (kswapd+0x854/0xdc0)
> [<800d8970>] (kswapd+0x854/0xdc0) from [<80051e28>] (kthread+0xc8/0xcc)
> [<80051e28>] (kthread+0xc8/0xcc) from [<80015198>] (ret_from_fork+0x14/0x20)


So the reclaim path seems to be marking UBIFS-backed pages as dirty
directly, I do not know why, the reclaim path is extremely complex and I
am no expert there. But may be someone on the MM list may help.

Note, this warning is not necessarily fatal. It just indicates that
UBIFS sees something which it believes should not happen.

> UBIFS assert failed in do_writepage at 936 (pid 543)
> CPU: 1 PID: 543 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #1
> [<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>] (show_stack+0x20/0x24)
> [<80019f44>] (show_stack+0x20/0x24) from [<80af2ef8>] (dump_stack+0x24/0x2c)
> [<80af2ef8>] (dump_stack+0x24/0x2c) from [<802990b8>] (do_writepage+0x1b8/0x1c4)
> [<802990b8>] (do_writepage+0x1b8/0x1c4) from [<802991e8>] (ubifs_writepage+0x124/0x1dc)
> [<802991e8>] (ubifs_writepage+0x124/0x1dc) from [<800d6eb8>] (shrink_page_list+0x6f0/0x884)
> [<800d6eb8>] (shrink_page_list+0x6f0/0x884) from [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8)
> [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8) from [<800d7c20>] (shrink_lruvec+0x3ac/0x524)
> [<800d7c20>] (shrink_lruvec+0x3ac/0x524) from [<800d8970>] (kswapd+0x854/0xdc0)
> [<800d8970>] (kswapd+0x854/0xdc0) from [<80051e28>] (kthread+0xc8/0xcc)
> [<80051e28>] (kthread+0xc8/0xcc) from [<80015198>] (ret_from_fork+0x14/0x20)

And here UBIFS sees a page being writted, but there is no budget
allocated for it, so the write may fail with -ENOSPC (no space), which
is not supposed to ever happen.

This is not necessarily fatal either, but indicates that UBIFS's
assumptions about how the system functions are wrong.

Now the question is: is it UBIFS which has incorrect assumptions, or
this is the Linux MM which is not doing the right thing? I do not know
the answer, let's see if the MM list may give us a clue.

Thanks!

RE: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Jijiagang]

Hi,
Thanks for your reply.

> first of all, what is your architecture? ARM? And how easily can you reproduce this? And can you try a kernel newer than 3.10?
The architecture is ARM. It can be reproduced in about 10 minutes by run Monkey test.
The test command is:
monkey -s 1000 --ignore-crashes --ignore-timeouts --ignore-security-exceptions --pct-trackball 0 --pct-nav 0 --pct-majornav 0 --pct-anyevent 0 -v -v -v --throttle 300 1200000000 > /sdcard/monkey.log 2>&1 &

We have searched commits in the linux stable git repository, but found no related commits.

Lowmomery killer starts often on android platform.

> Now the question is: is it UBIFS which has incorrect assumptions, or this is the Linux MM which is not doing the right thing? I do not know the answer, let's see if the MM list may give us a clue.
Please give us a hand, any reply will be appreciated. 

Thanks!

-----Original Message-----
From: Artem Bityutskiy [mailto:dedekind1@gmail.com] 
Sent: Monday, October 20, 2014 9:12 PM
To: Caizhiyong; linux-fsdevel@vger.kernel.org; linux-mm@kvack.org
Cc: Jijiagang; adrian.hunter@intel.com; linux-mtd@lists.infradead.org; Wanli (welly)
Subject: Re: UBIFS assert failed in ubifs_set_page_dirty at 1421

Hi,

first of all, what is your architecture? ARM? And how easily can you reproduce this? And can you try a kernel newer than 3.10?

And for fs-devel and mm people, here is the link to the original report:
http://lists.infradead.org/pipermail/linux-mtd/2014-October/055930.html, 

On Mon, 2014-10-20 at 12:01 +0000, Caizhiyong wrote:
> Here is part of the log, linux version 3.10:
>    cache 16240kB is below limit 16384kB for oom_score_adj 529
>    Free memory is -1820kB above reserved
> lowmemorykiller: Killing '.networkupgrade' (6924), adj 705,
>    to free 20968kB on behalf of 'kswapd0' (543) because
>    cache 16240kB is below limit 16384kB for oom_score_adj 529
>    Free memory is -2192kB above reserved

OK, no memory and OOM starts. So your system is in trouble anyway :-)

> UBIFS assert failed in ubifs_set_page_dirty at 1421 (pid 543)

UBIFS complain here that someone marks a page as dirty "directly", not through one of the UBIFS functions. And that someone is the page reclaim path.

Now, I do not really know what is going on here, so I am CCing a couple of mailing lists, may be someone will help.

Here is what I see is going on.

1. UBIFS wants to make sure that no one marks UBIFS-backed pages (and actually inodes too) as dirty directly. UBIFS wants everyone to ask UBIFS to mark a page as dirty.

2. This is because for every dirty page, UBIFS needs to reserve certain amount of space on the flash media, because all writes are out-of-place, even when you are changing an existing file.

3. There are exactly 2 places where UBIFS-backed pages may be marked as
dirty:

  a) ubifs_write_end() [->wirte_end] - the file write path
  b) ubifs_page_mkwrite() [->page_mkwirte] - the file mmap() path

4. If anything calls 'ubifs_set_page_dirty()' directly (not through write_end()/mkwrite()), and the page was not dirty, UBIFS will complain with the assertion that you see.

> CPU: 3 PID: 543 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #1
> [<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>] 
> (show_stack+0x20/0x24) [<80019f44>] (show_stack+0x20/0x24) from 
> [<80af2ef8>] (dump_stack+0x24/0x2c) [<80af2ef8>] 
> (dump_stack+0x24/0x2c) from [<80297234>] 
> (ubifs_set_page_dirty+0x54/0x5c) [<80297234>] 
> (ubifs_set_page_dirty+0x54/0x5c) from [<800cea60>] 
> (set_page_dirty+0x50/0x78) [<800cea60>] (set_page_dirty+0x50/0x78) 
> from [<800f4be4>] (try_to_unmap_one+0x1f8/0x3d0) [<800f4be4>] 
> (try_to_unmap_one+0x1f8/0x3d0) from [<800f4f44>] 
> (try_to_unmap_file+0x9c/0x740) [<800f4f44>] 
> (try_to_unmap_file+0x9c/0x740) from [<800f5678>] 
> (try_to_unmap+0x40/0x78) [<800f5678>] (try_to_unmap+0x40/0x78) from 
> [<800d6a04>] (shrink_page_list+0x23c/0x884) [<800d6a04>] 
> (shrink_page_list+0x23c/0x884) from [<800d76c8>] 
> (shrink_inactive_list+0x21c/0x3c8)
> [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8) from [<800d7c20>] 
> (shrink_lruvec+0x3ac/0x524) [<800d7c20>] (shrink_lruvec+0x3ac/0x524) 
> from [<800d8970>] (kswapd+0x854/0xdc0) [<800d8970>] 
> (kswapd+0x854/0xdc0) from [<80051e28>] (kthread+0xc8/0xcc) 
> [<80051e28>] (kthread+0xc8/0xcc) from [<80015198>] 
> (ret_from_fork+0x14/0x20)


So the reclaim path seems to be marking UBIFS-backed pages as dirty directly, I do not know why, the reclaim path is extremely complex and I am no expert there. But may be someone on the MM list may help.

Note, this warning is not necessarily fatal. It just indicates that UBIFS sees something which it believes should not happen.

> UBIFS assert failed in do_writepage at 936 (pid 543)
> CPU: 1 PID: 543 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #1
> [<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>] 
> (show_stack+0x20/0x24) [<80019f44>] (show_stack+0x20/0x24) from 
> [<80af2ef8>] (dump_stack+0x24/0x2c) [<80af2ef8>] 
> (dump_stack+0x24/0x2c) from [<802990b8>] (do_writepage+0x1b8/0x1c4) 
> [<802990b8>] (do_writepage+0x1b8/0x1c4) from [<802991e8>] 
> (ubifs_writepage+0x124/0x1dc) [<802991e8>] 
> (ubifs_writepage+0x124/0x1dc) from [<800d6eb8>] 
> (shrink_page_list+0x6f0/0x884) [<800d6eb8>] 
> (shrink_page_list+0x6f0/0x884) from [<800d76c8>] 
> (shrink_inactive_list+0x21c/0x3c8)
> [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8) from [<800d7c20>] 
> (shrink_lruvec+0x3ac/0x524) [<800d7c20>] (shrink_lruvec+0x3ac/0x524) 
> from [<800d8970>] (kswapd+0x854/0xdc0) [<800d8970>] 
> (kswapd+0x854/0xdc0) from [<80051e28>] (kthread+0xc8/0xcc) 
> [<80051e28>] (kthread+0xc8/0xcc) from [<80015198>] 
> (ret_from_fork+0x14/0x20)

And here UBIFS sees a page being writted, but there is no budget allocated for it, so the write may fail with -ENOSPC (no space), which is not supposed to ever happen.

This is not necessarily fatal either, but indicates that UBIFS's assumptions about how the system functions are wrong.

Now the question is: is it UBIFS which has incorrect assumptions, or this is the Linux MM which is not doing the right thing? I do not know the answer, let's see if the MM list may give us a clue.

Thanks!

Re: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Dave Chinner]

On Mon, Oct 20, 2014 at 04:11:59PM +0300, Artem Bityutskiy wrote:
> 3. There are exactly 2 places where UBIFS-backed pages may be marked as
> dirty:
> 
>   a) ubifs_write_end() [->wirte_end] - the file write path
>   b) ubifs_page_mkwrite() [->page_mkwirte] - the file mmap() path
> 
> 4. If anything calls 'ubifs_set_page_dirty()' directly (not through
> write_end()/mkwrite()), and the page was not dirty, UBIFS will complain
> with the assertion that you see.
> 
> > CPU: 3 PID: 543 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #1

Kernel is tainted. Not worth wasting time on unless it can be
reproduced on an untainted kernel...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

RE: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Jijiagang]

Dear Dave,
Thanks for your reply.
We removed some modules to keep kernel untainted. It still can be reproduced.

Here is the log:
UBIFS assert failed in ubifs_set_page_dirty at 1421 (pid 543)
CPU: 1 PID: 543 Comm: kswapd0 Not tainted 3.10.0_s40 #1
[<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>] (show_stack+0x20/0x24)
[<80019f44>] (show_stack+0x20/0x24) from [<80af2ef8>] (dump_stack+0x24/0x2c)
[<80af2ef8>] (dump_stack+0x24/0x2c) from [<80297234>] (ubifs_set_page_dirty+0x54/0x5c)
[<80297234>] (ubifs_set_page_dirty+0x54/0x5c) from [<800cea60>] (set_page_dirty+0x50/0x78)
[<800cea60>] (set_page_dirty+0x50/0x78) from [<800f4be4>] (try_to_unmap_one+0x1f8/0x3d0)
[<800f4be4>] (try_to_unmap_one+0x1f8/0x3d0) from [<800f4f44>] (try_to_unmap_file+0x9c/0x740)
[<800f4f44>] (try_to_unmap_file+0x9c/0x740) from [<800f5678>] (try_to_unmap+0x40/0x78)
[<800f5678>] (try_to_unmap+0x40/0x78) from [<800d6a04>] (shrink_page_list+0x23c/0x884)
[<800d6a04>] (shrink_page_list+0x23c/0x884) from [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8)
[<800d76c8>] (shrink_inactive_list+0x21c/0x3c8) from [<800d7c20>] (shrink_lruvec+0x3ac/0x524)
[<800d7c20>] (shrink_lruvec+0x3ac/0x524) from [<800d8970>] (kswapd+0x854/0xdc0)
[<800d8970>] (kswapd+0x854/0xdc0) from [<80051e28>] (kthread+0xc8/0xcc)
[<80051e28>] (kthread+0xc8/0xcc) from [<80015198>] (ret_from_fork+0x14/0x20)
UBIFS assert failed in ubifs_writepage at 1009 (pid 6)
CPU: 0 PID: 6 Comm: kworker/u8:0 Not tainted 3.10.0_s40 #1
Workqueue: writeback bdi_writeback_workfn (flush-ubifs_1_0)
[<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>] (show_stack+0x20/0x24)
[<80019f44>] (show_stack+0x20/0x24) from [<80af2ef8>] (dump_stack+0x24/0x2c)
[<80af2ef8>] (dump_stack+0x24/0x2c) from [<80299294>] (ubifs_writepage+0x1d0/0x1dc)
[<80299294>] (ubifs_writepage+0x1d0/0x1dc) from [<800cefc0>] (__writepage+0x24/0x4c)
[<800cefc0>] (__writepage+0x24/0x4c) from [<800cf288>] (write_cache_pages+0x1b0/0x408)
[<800cf288>] (write_cache_pages+0x1b0/0x408) from [<800cf538>] (generic_writepages+0x58/0x70)
[<800cf538>] (generic_writepages+0x58/0x70) from [<800cf594>] (do_writepages+0x44/0x48)
[<800cf594>] (do_writepages+0x44/0x48) from [<80139a30>] (__writeback_single_inode+0x50/0x238)
[<80139a30>] (__writeback_single_inode+0x50/0x238) from [<8013ae48>] (writeback_sb_inodes+0x264/0x44c)
[<8013ae48>] (writeback_sb_inodes+0x264/0x44c) from [<8013b0c4>] (__writeback_inodes_wb+0x94/0xcc)
[<8013b0c4>] (__writeback_inodes_wb+0x94/0xcc) from [<8013b3dc>] (wb_writeback+0x228/0x2f8)
[<8013b3dc>] (wb_writeback+0x228/0x2f8) from [<8013b6b4>] (wb_do_writeback+0x208/0x24c)
[<8013b6b4>] (wb_do_writeback+0x208/0x24c) from [<8013b774>] (bdi_writeback_workfn+0x7c/0x1dc)
[<8013b774>] (bdi_writeback_workfn+0x7c/0x1dc) from [<8004ac64>] (process_one_work+0x160/0x460)
[<8004ac64>] (process_one_work+0x160/0x460) from [<8004b0ac>] (worker_thread+0x148/0x49c)
[<8004b0ac>] (worker_thread+0x148/0x49c) from [<80051e28>] (kthread+0xc8/0xcc)
[<80051e28>] (kthread+0xc8/0xcc) from [<80015198>] (ret_from_fork+0x14/0x20)
UBIFS assert failed in do_writepage at 936 (pid 6)
CPU: 3 PID: 6 Comm: kworker/u8:0 Not tainted 3.10.0_s40 #1
Workqueue: writeback bdi_writeback_workfn (flush-ubifs_1_0)
[<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>] (show_stack+0x20/0x24)
[<80019f44>] (show_stack+0x20/0x24) from [<80af2ef8>] (dump_stack+0x24/0x2c)
[<80af2ef8>] (dump_stack+0x24/0x2c) from [<802990b8>] (do_writepage+0x1b8/0x1c4)
[<802990b8>] (do_writepage+0x1b8/0x1c4) from [<802991e8>] (ubifs_writepage+0x124/0x1dc)
[<802991e8>] (ubifs_writepage+0x124/0x1dc) from [<800cefc0>] (__writepage+0x24/0x4c)
[<800cefc0>] (__writepage+0x24/0x4c) from [<800cf288>] (write_cache_pages+0x1b0/0x408)
[<800cf288>] (write_cache_pages+0x1b0/0x408) from [<800cf538>] (generic_writepages+0x58/0x70)
[<800cf538>] (generic_writepages+0x58/0x70) from [<800cf594>] (do_writepages+0x44/0x48)
[<800cf594>] (do_writepages+0x44/0x48) from [<80139a30>] (__writeback_single_inode+0x50/0x238)
[<80139a30>] (__writeback_single_inode+0x50/0x238) from [<8013ae48>] (writeback_sb_inodes+0x264/0x44c)
[<8013ae48>] (writeback_sb_inodes+0x264/0x44c) from [<8013b0c4>] (__writeback_inodes_wb+0x94/0xcc)
[<8013b0c4>] (__writeback_inodes_wb+0x94/0xcc) from [<8013b3dc>] (wb_writeback+0x228/0x2f8)
[<8013b3dc>] (wb_writeback+0x228/0x2f8) from [<8013b6b4>] (wb_do_writeback+0x208/0x24c)
[<8013b6b4>] (wb_do_writeback+0x208/0x24c) from [<8013b774>] (bdi_writeback_workfn+0x7c/0x1dc)
[<8013b774>] (bdi_writeback_workfn+0x7c/0x1dc) from [<8004ac64>] (process_one_work+0x160/0x460)
[<8004ac64>] (process_one_work+0x160/0x460) from [<8004b0ac>] (worker_thread+0x148/0x49c)
[<8004b0ac>] (worker_thread+0x148/0x49c) from [<80051e28>] (kthread+0xc8/0xcc)
[<80051e28>] (kthread+0xc8/0xcc) from [<80015198>] (ret_from_fork+0x14/0x20)
UBIFS assert failed in ubifs_release_budget at 567 (pid 6)
CPU: 3 PID: 6 Comm: kworker/u8:0 Not tainted 3.10.0_s40 #1
Workqueue: writeback bdi_writeback_workfn (flush-ubifs_1_0)

-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Tuesday, October 21, 2014 11:39 AM
To: Artem Bityutskiy
Cc: Caizhiyong; linux-fsdevel@vger.kernel.org; linux-mm@kvack.org; Jijiagang; adrian.hunter@intel.com; linux-mtd@lists.infradead.org; Wanli (welly)
Subject: Re: UBIFS assert failed in ubifs_set_page_dirty at 1421

On Mon, Oct 20, 2014 at 04:11:59PM +0300, Artem Bityutskiy wrote:
> 3. There are exactly 2 places where UBIFS-backed pages may be marked 
> as
> dirty:
> 
>   a) ubifs_write_end() [->wirte_end] - the file write path
>   b) ubifs_page_mkwrite() [->page_mkwirte] - the file mmap() path
> 
> 4. If anything calls 'ubifs_set_page_dirty()' directly (not through 
> write_end()/mkwrite()), and the page was not dirty, UBIFS will 
> complain with the assertion that you see.
> 
> > CPU: 3 PID: 543 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #1

Kernel is tainted. Not worth wasting time on unless it can be reproduced on an untainted kernel...

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

RE: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Jijiagang]

Dears,
For this problem, we find it relates to MM migration.
If disable migration it will not happen.

The call sequence is:
alloc_contig_range (mm/page_alloc.c)
 -> __alloc_contig_migrate_range (mm/page_alloc.c)
  ->migrate_pages (mm/migrate.c)
   ->try_to_unmap (mm/rmap.c)
    ->try_to_unmap_file (mm/rmap.c)
     ->try_to_unmap_one (mm/rmap.c)
      ->set_page_dirty (mm/page-writeback.c)
       ->ubifs_set_page_dirty (fs/ubifs/file.c)
The ubifs_set_page_dirty is provided in ubifs, it will be called out of ubifs.
And it just migrate file page to another page in memory, not in flash, it needn't to budget for it.

The question is:
1. Does ubifs support migrate or not?
2. Or mm not do the right thing?

Could you please give us a hand to solve it? Any reply will be appreciated. Thanks!

Best Regards.

> -----Original Message-----
> From: Artem Bityutskiy [mailto:dedekind1@gmail.com]
> Sent: Monday, October 20, 2014 9:12 PM
> To: Caizhiyong; linux-fsdevel@vger.kernel.org; linux-mm@kvack.org
> Cc: Jijiagang; adrian.hunter@intel.com; linux-mtd@lists.infradead.org; Wanli
> (welly)
> Subject: Re: UBIFS assert failed in ubifs_set_page_dirty at 1421
> 
> Hi,
> 
> first of all, what is your architecture? ARM? And how easily can you reproduce
> this? And can you try a kernel newer than 3.10?
> 
> And for fs-devel and mm people, here is the link to the original report:
> http://lists.infradead.org/pipermail/linux-mtd/2014-October/055930.html,
> 
> On Mon, 2014-10-20 at 12:01 +0000, Caizhiyong wrote:
> > Here is part of the log, linux version 3.10:
> >    cache 16240kB is below limit 16384kB for oom_score_adj 529
> >    Free memory is -1820kB above reserved
> > lowmemorykiller: Killing '.networkupgrade' (6924), adj 705,
> >    to free 20968kB on behalf of 'kswapd0' (543) because
> >    cache 16240kB is below limit 16384kB for oom_score_adj 529
> >    Free memory is -2192kB above reserved
> 
> OK, no memory and OOM starts. So your system is in trouble anyway :-)
> 
> > UBIFS assert failed in ubifs_set_page_dirty at 1421 (pid 543)
> 
> UBIFS complain here that someone marks a page as dirty "directly", not
> through one of the UBIFS functions. And that someone is the page reclaim path.
> 
> Now, I do not really know what is going on here, so I am CCing a couple of
> mailing lists, may be someone will help.
> 
> Here is what I see is going on.
> 
> 1. UBIFS wants to make sure that no one marks UBIFS-backed pages (and
> actually inodes too) as dirty directly. UBIFS wants everyone to ask UBIFS to mark
> a page as dirty.
> 
> 2. This is because for every dirty page, UBIFS needs to reserve certain amount of
> space on the flash media, because all writes are out-of-place, even when you
> are changing an existing file.
> 
> 3. There are exactly 2 places where UBIFS-backed pages may be marked as
> dirty:
> 
>   a) ubifs_write_end() [->wirte_end] - the file write path
>   b) ubifs_page_mkwrite() [->page_mkwirte] - the file mmap() path
> 
> 4. If anything calls 'ubifs_set_page_dirty()' directly (not through
> write_end()/mkwrite()), and the page was not dirty, UBIFS will complain with
> the assertion that you see.
> 
> > CPU: 3 PID: 543 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #1
> > [<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>]
> > (show_stack+0x20/0x24) [<80019f44>] (show_stack+0x20/0x24) from
> > [<80af2ef8>] (dump_stack+0x24/0x2c) [<80af2ef8>]
> > (dump_stack+0x24/0x2c) from [<80297234>]
> > (ubifs_set_page_dirty+0x54/0x5c) [<80297234>]
> > (ubifs_set_page_dirty+0x54/0x5c) from [<800cea60>]
> > (set_page_dirty+0x50/0x78) [<800cea60>] (set_page_dirty+0x50/0x78)
> > from [<800f4be4>] (try_to_unmap_one+0x1f8/0x3d0) [<800f4be4>]
> > (try_to_unmap_one+0x1f8/0x3d0) from [<800f4f44>]
> > (try_to_unmap_file+0x9c/0x740) [<800f4f44>]
> > (try_to_unmap_file+0x9c/0x740) from [<800f5678>]
> > (try_to_unmap+0x40/0x78) [<800f5678>] (try_to_unmap+0x40/0x78) from
> > [<800d6a04>] (shrink_page_list+0x23c/0x884) [<800d6a04>]
> > (shrink_page_list+0x23c/0x884) from [<800d76c8>]
> > (shrink_inactive_list+0x21c/0x3c8)
> > [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8) from [<800d7c20>]
> > (shrink_lruvec+0x3ac/0x524) [<800d7c20>] (shrink_lruvec+0x3ac/0x524)
> > from [<800d8970>] (kswapd+0x854/0xdc0) [<800d8970>]
> > (kswapd+0x854/0xdc0) from [<80051e28>] (kthread+0xc8/0xcc)
> > [<80051e28>] (kthread+0xc8/0xcc) from [<80015198>]
> > (ret_from_fork+0x14/0x20)
> 
> 
> So the reclaim path seems to be marking UBIFS-backed pages as dirty directly, I
> do not know why, the reclaim path is extremely complex and I am no expert
> there. But may be someone on the MM list may help.
> 
> Note, this warning is not necessarily fatal. It just indicates that UBIFS sees
> something which it believes should not happen.
> 
> > UBIFS assert failed in do_writepage at 936 (pid 543)
> > CPU: 1 PID: 543 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #1
> > [<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>]
> > (show_stack+0x20/0x24) [<80019f44>] (show_stack+0x20/0x24) from
> > [<80af2ef8>] (dump_stack+0x24/0x2c) [<80af2ef8>]
> > (dump_stack+0x24/0x2c) from [<802990b8>] (do_writepage+0x1b8/0x1c4)
> > [<802990b8>] (do_writepage+0x1b8/0x1c4) from [<802991e8>]
> > (ubifs_writepage+0x124/0x1dc) [<802991e8>]
> > (ubifs_writepage+0x124/0x1dc) from [<800d6eb8>]
> > (shrink_page_list+0x6f0/0x884) [<800d6eb8>]
> > (shrink_page_list+0x6f0/0x884) from [<800d76c8>]
> > (shrink_inactive_list+0x21c/0x3c8)
> > [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8) from [<800d7c20>]
> > (shrink_lruvec+0x3ac/0x524) [<800d7c20>] (shrink_lruvec+0x3ac/0x524)
> > from [<800d8970>] (kswapd+0x854/0xdc0) [<800d8970>]
> > (kswapd+0x854/0xdc0) from [<80051e28>] (kthread+0xc8/0xcc)
> > [<80051e28>] (kthread+0xc8/0xcc) from [<80015198>]
> > (ret_from_fork+0x14/0x20)
> 
> And here UBIFS sees a page being writted, but there is no budget allocated for
> it, so the write may fail with -ENOSPC (no space), which is not supposed to ever
> happen.
> 
> This is not necessarily fatal either, but indicates that UBIFS's assumptions about
> how the system functions are wrong.
> 
> Now the question is: is it UBIFS which has incorrect assumptions, or this is the
> Linux MM which is not doing the right thing? I do not know the answer, let's see
> if the MM list may give us a clue.
> 
> Thanks!

Re: UBIFS assert failed in ubifs_set_page_dirty at 1421

[hujianyang]

Hi,

I think we found the cause of this problem. We enable CONFIG_CMA in our
config file. This feature seems to allocate a contiguous memory for caller.
If some pages in this contiguous area are used by other modules, like UBIFS,
CMA will migrate these pages to other place. This operation should be
transparent to the user of old pages. But it is *not true* for UBIFS.

> 
> 1. UBIFS wants to make sure that no one marks UBIFS-backed pages (and
> actually inodes too) as dirty directly. UBIFS wants everyone to ask
> UBIFS to mark a page as dirty.
> 
> 2. This is because for every dirty page, UBIFS needs to reserve certain
> amount of space on the flash media, because all writes are out-of-place,
> even when you are changing an existing file.
> 
> 3. There are exactly 2 places where UBIFS-backed pages may be marked as
> dirty:
> 
>   a) ubifs_write_end() [->wirte_end] - the file write path
>   b) ubifs_page_mkwrite() [->page_mkwirte] - the file mmap() path

line 1160, func try_to_unmap_one() in mm/rmap.c

""
        /* Move the dirty bit to the physical page now the pte is gone. */
        if (pte_dirty(pteval))
                set_page_dirty(page);
""

Here, If the pte of a page is dirty, a directly set_page_dirty() is
performed and hurt the internal logic of UBIFS.

So I have a question, why the page needs to be marked as dirty when
the pte is dirty? Can we just move the dirty bit of the old pte to the
new one without setting the dirty bit of the page? I think the dirty
bit of a page is used to mark the contents of this page is different
from it is in storage. Can we just set it without informing filesyetem?

Could any one in MM list show us some reasons of this performing or
give us some help?

> 
> 4. If anything calls 'ubifs_set_page_dirty()' directly (not through
> write_end()/mkwrite()), and the page was not dirty, UBIFS will complain
> with the assertion that you see.
> 

To Artem, I have an idea to fix this problem without changing mm files.
We can:

1) Add a .migratepage in address_space_operations and do budget in it.
2) As CMA perform set_page_dirty() before migratepage, we seems to remove
   the assert failed in ubifs_set_page_dirty().

Just some thoughts, not a clear solution.

I should say I don't want to do this because it will change some logic
in UBIFS. So I think the best way to solve this problem is to change
the MM operations.

Thanks!

Hu



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Kirill A. Shutemov]

On Fri, Nov 07, 2014 at 10:22:38AM +0800, hujianyang wrote:
> Hi,
> 
> I think we found the cause of this problem. We enable CONFIG_CMA in our
> config file. This feature seems to allocate a contiguous memory for caller.
> If some pages in this contiguous area are used by other modules, like UBIFS,
> CMA will migrate these pages to other place. This operation should be
> transparent to the user of old pages. But it is *not true* for UBIFS.
> 
> > 
> > 1. UBIFS wants to make sure that no one marks UBIFS-backed pages (and
> > actually inodes too) as dirty directly. UBIFS wants everyone to ask
> > UBIFS to mark a page as dirty.
> > 
> > 2. This is because for every dirty page, UBIFS needs to reserve certain
> > amount of space on the flash media, because all writes are out-of-place,
> > even when you are changing an existing file.
> > 
> > 3. There are exactly 2 places where UBIFS-backed pages may be marked as
> > dirty:
> > 
> >   a) ubifs_write_end() [->wirte_end] - the file write path
> >   b) ubifs_page_mkwrite() [->page_mkwirte] - the file mmap() path
> 
> line 1160, func try_to_unmap_one() in mm/rmap.c
> 
> ""
>         /* Move the dirty bit to the physical page now the pte is gone. */
>         if (pte_dirty(pteval))
>                 set_page_dirty(page);
> ""
> 
> Here, If the pte of a page is dirty, a directly set_page_dirty() is
> performed and hurt the internal logic of UBIFS.

If the pte is dirty it must be writable too. And to make pte writable
->page_mkwrite() must be called. So it should work fine..

Could you check if the pteval is pte_write() by the time?
And could you provide what says dump_page(page) and dump_vma(vma) while
you are there?

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Jijiagang]

Hi Kirill,

I add dump_page(page) in function ubifs_set_page_dirty.
And get this log when ubifs assert fail. Is it helpful for this problem?

page:81411740 count:3 mapcount:1 mapping:a33db634 index:0x0
page flags: 0x219(locked|uptodate|dirty|arch_1)

UBIFS assert failed in ubifs_set_page_dirty at 1424 (pid 545)
CPU: 1 PID: 545 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #18
[<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>] (show_stack+0x20/0x24)
[<80019f44>] (show_stack+0x20/0x24) from [<80acf9f8>] (dump_stack+0x24/0x2c)
[<80acf9f8>] (dump_stack+0x24/0x2c) from [<80298870>] (ubifs_set_page_dirty+0x60/0x6c)
[<80298870>] (ubifs_set_page_dirty+0x60/0x6c) from [<800cea60>] (set_page_dirty+0x50/0x78)
[<800cea60>] (set_page_dirty+0x50/0x78) from [<800f4be4>] (try_to_unmap_one+0x1f8/0x3d0)
[<800f4be4>] (try_to_unmap_one+0x1f8/0x3d0) from [<800f4f44>] (try_to_unmap_file+0x9c/0x740)
[<800f4f44>] (try_to_unmap_file+0x9c/0x740) from [<800f5678>] (try_to_unmap+0x40/0x78)
[<800f5678>] (try_to_unmap+0x40/0x78) from [<800d6a04>] (shrink_page_list+0x23c/0x884)
[<800d6a04>] (shrink_page_list+0x23c/0x884) from [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8)
[<800d76c8>] (shrink_inactive_list+0x21c/0x3c8) from [<800d7c20>] (shrink_lruvec+0x3ac/0x524)
[<800d7c20>] (shrink_lruvec+0x3ac/0x524) from [<800d8970>] (kswapd+0x854/0xdc0)
[<800d8970>] (kswapd+0x854/0xdc0) from [<80051e28>] (kthread+0xc8/0xcc)
[<80051e28>] (kthread+0xc8/0xcc) from [<80015198>] (ret_from_fork+0x14/0x20)

> -----Original Message-----
> From: Kirill A. Shutemov [mailto:kirill@shutemov.name]
> Sent: Thursday, November 20, 2014 8:30 PM
> To: Hujianyang
> Cc: dedekind1@gmail.com; Caizhiyong; linux-fsdevel@vger.kernel.org;
> linux-mm@kvack.org; Wanli (welly); linux-mtd@lists.infradead.org;
> adrian.hunter@intel.com; Jijiagang
> Subject: Re: UBIFS assert failed in ubifs_set_page_dirty at 1421
> 
> On Fri, Nov 07, 2014 at 10:22:38AM +0800, hujianyang wrote:
> > Hi,
> >
> > I think we found the cause of this problem. We enable CONFIG_CMA in
> > our config file. This feature seems to allocate a contiguous memory for caller.
> > If some pages in this contiguous area are used by other modules, like
> > UBIFS, CMA will migrate these pages to other place. This operation
> > should be transparent to the user of old pages. But it is *not true* for UBIFS.
> >
> > >
> > > 1. UBIFS wants to make sure that no one marks UBIFS-backed pages
> > > (and actually inodes too) as dirty directly. UBIFS wants everyone to
> > > ask UBIFS to mark a page as dirty.
> > >
> > > 2. This is because for every dirty page, UBIFS needs to reserve
> > > certain amount of space on the flash media, because all writes are
> > > out-of-place, even when you are changing an existing file.
> > >
> > > 3. There are exactly 2 places where UBIFS-backed pages may be marked
> > > as
> > > dirty:
> > >
> > >   a) ubifs_write_end() [->wirte_end] - the file write path
> > >   b) ubifs_page_mkwrite() [->page_mkwirte] - the file mmap() path
> >
> > line 1160, func try_to_unmap_one() in mm/rmap.c
> >
> > ""
> >         /* Move the dirty bit to the physical page now the pte is gone. */
> >         if (pte_dirty(pteval))
> >                 set_page_dirty(page);
> > ""
> >
> > Here, If the pte of a page is dirty, a directly set_page_dirty() is
> > performed and hurt the internal logic of UBIFS.
> 
> If the pte is dirty it must be writable too. And to make pte writable
> ->page_mkwrite() must be called. So it should work fine..
> 
> Could you check if the pteval is pte_write() by the time?
> And could you provide what says dump_page(page) and dump_vma(vma) while
> you are there?
> 
> --
>  Kirill A. Shutemov

Re: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Kirill A. Shutemov]

On Mon, Nov 24, 2014 at 02:59:51AM +0000, Jijiagang wrote:
> Hi Kirill,
> 
> I add dump_page(page) in function ubifs_set_page_dirty.
> And get this log when ubifs assert fail. Is it helpful for this problem?

Not really. It seems you called dump_page() after
__set_page_dirty_nobuffers() in ubifs_set_page_dirty().

Could you try something like patch below. It assumes ubifs to compiled in
(not module).

diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c
index b5b593c45270..7b4386dd174e 100644
--- a/fs/ubifs/file.c
+++ b/fs/ubifs/file.c
@@ -1531,7 +1531,7 @@ out_unlock:
        return err;
 }
 
-static const struct vm_operations_struct ubifs_file_vm_ops = {
+const struct vm_operations_struct ubifs_file_vm_ops = {
        .fault        = filemap_fault,
        .map_pages = filemap_map_pages,
        .page_mkwrite = ubifs_vm_page_mkwrite,
diff --git a/mm/rmap.c b/mm/rmap.c
index 19886fb2f13a..343c4571df68 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1171,8 +1171,15 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
        pteval = ptep_clear_flush(vma, address, pte);
 
        /* Move the dirty bit to the physical page now the pte is gone. */
-       if (pte_dirty(pteval))
+       if (pte_dirty(pteval)) {
+               extern const struct vm_operations_struct ubifs_file_vm_ops;
+               if (vma->vm_ops == &ubifs_file_vm_ops) {
+                       dump_vma(vma);
+                       dump_page(page, __func__);
+                       pr_emerg("pte_write: %d\n", pte_write(pteval));
+               }
                set_page_dirty(page);
+       }
 
        /* Update high watermark before we lower rss */
        update_hiwater_rss(mm);
-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Jijiagang]

Hi Kirill,
I test your patch, bus there's no dump_vma definition.
The log is here, hope it will be helpful.

 page:817fd7e0 count:3 mapcount:1 mapping:a318bb8c index:0x4
 page flags: 0xa19(locked|uptodate|dirty|arch_1|private)
 pte_write: 1
 page:81441a80 count:3 mapcount:1 mapping:a318bb8c index:0x5
 page flags: 0x209(locked|uptodate|arch_1)
 pte_write: 1
 UBIFS assert failed in ubifs_set_page_dirty at 1422 (pid 545)
 CPU: 2 PID: 545 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #19
 [<8001d8a0>] (unwind_backtrace+0x0/0x108) from [<80019f44>] (show_stack+0x20/0x24)
 [<80019f44>] (show_stack+0x20/0x24) from [<80acfa18>] (dump_stack+0x24/0x2c)
 [<80acfa18>] (dump_stack+0x24/0x2c) from [<8029766c>] (ubifs_set_page_dirty+0x54/0x5c)
 [<8029766c>] (ubifs_set_page_dirty+0x54/0x5c) from [<800cea60>] (set_page_dirty+0x50/0x78)
 [<800cea60>] (set_page_dirty+0x50/0x78) from [<800f4b10>] (try_to_unmap_one+0x124/0x410)
 [<800f4b10>] (try_to_unmap_one+0x124/0x410) from [<800f4f84>] (try_to_unmap_file+0x9c/0x740)
 [<800f4f84>] (try_to_unmap_file+0x9c/0x740) from [<800f56b8>] (try_to_unmap+0x40/0x78)
 [<800f56b8>] (try_to_unmap+0x40/0x78) from [<800d6a04>] (shrink_page_list+0x23c/0x884)
 [<800d6a04>] (shrink_page_list+0x23c/0x884) from [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8)
 [<800d76c8>] (shrink_inactive_list+0x21c/0x3c8) from [<800d7c20>] (shrink_lruvec+0x3ac/0x524)
 [<800d7c20>] (shrink_lruvec+0x3ac/0x524) from [<800d8970>] (kswapd+0x854/0xdc0)
 [<800d8970>] (kswapd+0x854/0xdc0) from [<80051e28>] (kthread+0xc8/0xcc)
 [<80051e28>] (kthread+0xc8/0xcc) from [<80015198>] (ret_from_fork+0x14/0x20)
 UBIFS assert failed in ubifs_release_budget at 567 (pid 6)
 CPU: 3 PID: 6 Comm: kworker/u8:0 Tainted: P           O 3.10.0_s40 #19

> -----Original Message-----
> From: Kirill A. Shutemov [mailto:kirill@shutemov.name]
> Sent: Monday, November 24, 2014 5:10 PM
> To: Jijiagang
> Cc: Hujianyang; dedekind1@gmail.com; Caizhiyong;
> linux-fsdevel@vger.kernel.org; linux-mm@kvack.org; Wanli (welly);
> linux-mtd@lists.infradead.org; adrian.hunter@intel.com
> Subject: Re: UBIFS assert failed in ubifs_set_page_dirty at 1421
> 
> On Mon, Nov 24, 2014 at 02:59:51AM +0000, Jijiagang wrote:
> > Hi Kirill,
> >
> > I add dump_page(page) in function ubifs_set_page_dirty.
> > And get this log when ubifs assert fail. Is it helpful for this problem?
> 
> Not really. It seems you called dump_page() after
> __set_page_dirty_nobuffers() in ubifs_set_page_dirty().
> 
> Could you try something like patch below. It assumes ubifs to compiled in (not
> module).
> 
> diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c index b5b593c45270..7b4386dd174e
> 100644
> --- a/fs/ubifs/file.c
> +++ b/fs/ubifs/file.c
> @@ -1531,7 +1531,7 @@ out_unlock:
>         return err;
>  }
> 
> -static const struct vm_operations_struct ubifs_file_vm_ops = {
> +const struct vm_operations_struct ubifs_file_vm_ops = {
>         .fault        = filemap_fault,
>         .map_pages = filemap_map_pages,
>         .page_mkwrite = ubifs_vm_page_mkwrite, diff --git a/mm/rmap.c
> b/mm/rmap.c index 19886fb2f13a..343c4571df68 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
> @@ -1171,8 +1171,15 @@ static int try_to_unmap_one(struct page *page,
> struct vm_area_struct *vma,
>         pteval = ptep_clear_flush(vma, address, pte);
> 
>         /* Move the dirty bit to the physical page now the pte is gone. */
> -       if (pte_dirty(pteval))
> +       if (pte_dirty(pteval)) {
> +               extern const struct vm_operations_struct
> ubifs_file_vm_ops;
> +               if (vma->vm_ops == &ubifs_file_vm_ops) {
> +                       dump_vma(vma);
> +                       dump_page(page, __func__);
> +                       pr_emerg("pte_write: %d\n", pte_write(pteval));
> +               }
>                 set_page_dirty(page);
> +       }
> 
>         /* Update high watermark before we lower rss */
>         update_hiwater_rss(mm);
> --
>  Kirill A. Shutemov

Re: UBIFS assert failed in ubifs_set_page_dirty at 1421

[Kirill A. Shutemov]

On Mon, Nov 24, 2014 at 10:20:59AM +0000, Jijiagang wrote:
> Hi Kirill,
> I test your patch, bus there's no dump_vma definition.
> The log is here, hope it will be helpful.
> 
>  page:817fd7e0 count:3 mapcount:1 mapping:a318bb8c index:0x4
>  page flags: 0xa19(locked|uptodate|dirty|arch_1|private)
>  pte_write: 1
>  page:81441a80 count:3 mapcount:1 mapping:a318bb8c index:0x5
>  page flags: 0x209(locked|uptodate|arch_1)
>  pte_write: 1

Okay, pte is writable but page is not dirty and doesn't have private flag
set.

I looked through code and don't see how that could happen.

>  UBIFS assert failed in ubifs_set_page_dirty at 1422 (pid 545)
>  CPU: 2 PID: 545 Comm: kswapd0 Tainted: P           O 3.10.0_s40 #19

Is it possible to reproduce the issue on current upstream without any
patches or third-party modules?

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>