Re: [RFC][PATCH] Re: [BUG] ext4: cannot unfreeze a filesystem due to a deadlock

[Toshiyuki Okajima <toshi.okajima@jp.fujitsu.com>]

Hi,

(2011/04/18 19:51), Jan Kara wrote:
> On Mon 18-04-11 18:05:01, Toshiyuki Okajima wrote:
>>> On Fri 15-04-11 22:39:07, Toshiyuki Okajima wrote:
>>>>>    For ext3 or ext4 without delayed allocation we block inside writepage()
>>>>> function. But as I wrote to Dave Chinner, ->page_mkwrite() should probably
>>>>> get modified to block while minor-faulting the page on frozen fs because
>>>>> when blocks are already allocated we may skip starting a transaction and so
>>>>> we could possibly modify the filesystem.
>>>> OK. I think ->page_mkwrite() should also block writing the minor-faulting pages.
>>>>
>>>> (minor-pagefault)
>>>> ->   do_wp_page()
>>>>     ->   page_mkwrite(= ext4_mkwrite())
>>>>        =>   BLOCK!
>>>>
>>>> (major-pagefault)
>>>> ->   do_liner_fault()
>>>>     ->   page_mkwrite(= ext4_mkwrite())
>>>>        =>   BLOCK!
>>>>
>>>>>
>>>>>>>> Mizuma-san's reproducer also writes the data which maps to the file (mmap).
>>>>>>>> The original problem happens after the fsfreeze operation is done.
>>>>>>>> I understand the normal write operation (not mmap) can be blocked while
>>>>>>>> fsfreezing. So, I guess we don't always block all the write operation
>>>>>>>> while fsfreezing.
>>>>>>>    Technically speaking, we block all the transaction starts which means we
>>>>>>> end up blocking all the writes from going to disk. But that does not mean
>>>>>>> we block all the writes from going to in-memory cache - as you properly
>>>>>>> note the mmap case is one of such exceptions.
>>>>>> Hm, I also think we can allow the writes to in-memory cache but we can't allow
>>>>>> the writes to disk while fsfreezing. I am considering that mmap path can
>>>>>> write to disk while fsfreezing because this deadlock problem happens after
>>>>>> fsfreeze operation is done...
>>>>>    I'm sorry I don't understand now - are you speaking about the case above
>>>>> when writepage() does not wait for filesystem being frozen or something
>>>>> else?
>>>> Sorry, I didn't understand around the page fault path.
>>>> So, I had read the kernel source code around it, then I maybe understand...
>>>>
>>>> I worry whether we can update the file data in mmap case while fsfreezing.
>>>> Of course, I understand that we can write to in-memory cache, and it is not a
>>>> problem. However, if we can write to disk while fsfreezing, it is a problem.
>>>> So, I summarize the cases whether we can write to disk or not.
>>>>
>>>> --------------------------------------------------------------------------
>>>> Cases (Whether we can write the data mmapped to the file on the disk
>>>> while fsfreezing)
>>>>
>>>> [1] One of the page which has been mmapped is not bound. And
>>>>   the page is not allocated yet. (major fault?)
>>>>
>>>>     (1) user dirtys a page
>>>>     (2) a page fault occurs (do_page_fault)
>>>>     (3) __do_falut is called.
>>>>     (4) ext4_page_mkwrite is called
>>>>     (5) ext4_write_begin is called
>>>>     (6) ext4_journal_start_sb       =>   We can STOP!
>>>>
>>>> [2] One of the page which has been mmapped is not bound. But
>>>>   the page is already allocated, and the buffer_heads of the page
>>>>   are not mapped (BH_Mapped).  (minor fault?)
>>>>
>>>>     (1) user dirtys a page
>>>>     (2) a page fault occurs (do_page_fault)
>>>>     (3) do_wp_page is called.
>>>>     (4) ext4_page_mkwrite is called
>>>>     (5) ext4_write_begin is called
>>>>     (6) ext4_journal_start_sb       =>   We can STOP!
>>>>
>>>> [3] One of the page which has been mmapped is not bound. But
>>>>   the page is already allocated, and the buffer_heads of the page
>>>>   are mapped (BH_Mapped).  (minor fault?)
>>>>
>>>>     (1) user dirtys a page
>>>>     (2) a page fault occurs (do_page_fault)
>>>>     (3) do_wp_page is called.
>>>>     (4) ext4_page_mkwrite is called
>>>>     * Cannot block the dirty page to be written because all bh is mapped.
>>>>     (5) user munmaps the page (munmap)
>>>>     (6) zap_pte_range dirtys the page (struct page) which is pte_dirtyed.
>>>>     (7) writeback thread writes the page (struct page) to disk
>>>>                                             =>   We cannot STOP!
>>>>
>>>> [4] One of the page which has been mmapped is bound. And
>>>>   the page is already allocated.
>>>>
>>>>     (1) user dirtys a page
>>>>     ( ) no page fault occurs
>>>>     (2) user munmaps the page (munmap)
>>>>     (3) zap_pte_range dirtys the page (struct page) which is pte_dirtyed.
>>>>     (4) writeback thread writes the page (struct page) to disk
>>>>                                             =>   We cannot STOP!
>>>> --------------------------------------------------------------------------
>>>>
>>>> So, we can block the cases [1], [2].
>>>> But I think we cannot block the cases [3], [4] now.
>>>> If fixing the page_mkwrite, we can also block the case [3].
>>>> But the case [4] is not blocked because no page fault occurs
>>>> when we dirty the mmapped page.
>>>>
>>>> Therefore, to repair this problem, we need to fix the cases [3], [4].
>>>> I think we must modify the writeback thread to fix the case [4].
>>>    The trick here is that when we write a page to disk, we write-protect
>>> the page (you seem to call this that "the page is bound", I'm not sure why).
>> Hm, I want to understand how to write-protect the page under fsfreezing.
>    Look at what page_mkclean() called from clear_page_dirty_for_io() does...
Thanks. I'll read that.

>
>> But, anyway, I understand we don't need to consider the case [4].
>    Yes.
>
>>> So we are guaranteed to receive a minor fault (case [3]) if user tries to
>>> modify a page after we finish writeback while freezing the filesystem.
>>> So principially all we need to do is just wait in ext4_page_mkwrite().
>> OK. I understand.
>> Are there any concrete ideas to fix this?
>> For ext4, we can rescue from the case [3] by modifying ext4_page_mkwrite().
>    Yes.
>
>> But for ext3 or other FSs, we must implement ->page_mkwrite() to prevent it?
>    Sadly I don't see a simple way to fix this issue for all filesystems at
> once. Implementing proper wait in block_page_mkwrite() should fix the issue
> for xfs. Other filesystems like GFS2 or Btrfs will have to be fixed
> separately as ext4. For ext3, we'd have to add ->page_mkwrite() support. I
> have patches for this already for some time but I have to get to properly
> testing them in more exotic conditions like 64k pages...
OK. I understand the current status of your works to fix the problem which
can be written with some data at mmap path while fsfreezing.

Thanks,
Toshiyuki Okajima