[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688





------- Comment #2 from tytso@mit.edu  2008-11-06 12:27 -------
Can you replicate this on a newer kernel?  This looks like a failure in
ext3/jbd code, actually, triggered when the kernel was trying to free pages to
get back some memory.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688





------- Comment #3 from tytso@mit.edu  2009-01-17 16:24 -------
Hi, any more information about this bug? How much memory do you have?

What ext3 filesystems do you have mounted?  The oops is in ext3 code:


handle_t *journal_start(journal_t *journal, int nblocks)
{
        handle_t *handle = journal_current_handle();
        int err;

        if (!journal)
                return ERR_PTR(-EROFS);

        if (handle) {
                J_ASSERT(handle->h_transaction->t_journal == journal); <----
                handle->h_ref++;
                return handle;
        }

It seems to be related ext3's quota code -- and again it seems to be related to
an attempt to free memory when the system is under memory pressure.  This looks
very much like a quota and/or ext3 bug.




-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688





------- Comment #4 from vedranf@vedranf.mine.nu  2009-01-18 15:39 -------
Sure, I'm waiting for 2.6.28 to enter debian so I can test. I have 1.7GB of
memory, of that more than 1GB is free (excluding cache). All other filesystems
are ext3 including the source (ext3->ext4). Quota is enabled but not set for
any user. I disabled it (init.d/quota stop) and tried writing (ext3->ext4)
again and oops didn't happen. Will test more, but this could be the issue. What
I don't understand is why it doesn't happen when I'm writing to ext3.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688





------- Comment #5 from tytso@mit.edu  2009-01-18 16:56 -------
Ext4's delayed allocation will cause more memory pressure, because blocks
aren't as aggressively pushed out to disk.   How much memory do you have in
your system, how many ext3/ext4 filesystems do you have mounted, how many of
them are active (and how active) versus totally quiescent, and how big is the
journal for those filesystems?


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688





------- Comment #6 from vedranf@vedranf.mine.nu  2009-01-18 17:18 -------
                total       used       free     shared    buffers     cached
Mem:       1773      1730       43        0            23           1185
-/+ buffers/cache:   521       1252
Swap:            0          0          0

/dev/mapper/vg-root on / type ext3 (rw,errors=remount-ro)
/dev/mapper/vg-home on /home type ext3 (rw,usrquota)
/dev/mapper/vg-var on /var type ext3 (rw)
/dev/sda3 on /media/disk type ext4dev (rw,nosuid,nodev,uhelper=hal)
/dev/sda2 on /media/disk-1 type ext3 (rw,nosuid,nodev,uhelper=hal,data=ordered)

Copying from /home to /media/disk so only these are active if I understand you
correctly. 

# debugfs -R "stat <8>" /dev/mapper/vg-home                               
debugfs 1.41.3 (12-Oct-2008)
Inode: 8   Type: regular    Mode:  0600   Flags: 0x0
Generation: 0    Version: 0x00000000
User:     0   Group:     0   Size: 134217728
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 262416
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x487a96ce -- Mon Jul 14 01:59:10 2008
atime: 0x00000000 -- Thu Jan  1 01:00:00 1970
mtime: 0x487a96ce -- Mon Jul 14 01:59:10 2008
Size of extra inode fields: 0

# debugfs -R "stat <8>" /dev/sda3                                           
debugfs 1.41.3 (12-Oct-2008)
Inode: 8   Type: regular    Mode:  0600   Flags: 0x80000
Generation: 0    Version: 0x00000000
User:     0   Group:     0   Size: 134217728
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 262144
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x48ee6d44 -- Thu Oct  9 22:44:52 2008
atime: 0x00000000 -- Thu Jan  1 01:00:00 1970
mtime: 0x48ee6d44 -- Thu Oct  9 22:44:52 2008
Size of extra inode fields: 0


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688


tytso@mit.edu changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|fs_ext4@kernel-bugs.osdl.org|fs_ext3@kernel-bugs.osdl.org
          Component|ext4                        |ext3




------- Comment #7 from tytso@mit.edu  2009-01-18 18:28 -------
So you only have 2 megabytes of memory....  Yeah, this sounds like a low-memory
pressure situation.   Ext4's delayed allocation will definitely use more
memory, and it looks like you are tickling an ext3 memory pressure bug.   Is
there a reason why you aren't mounting your filesystems with noatime?   That's
undoubtedly responsible for the ext3 journal activity, and I suspect mounting
/home with noatime will make the problem go away.  It might also be interesting
disabling quotas on /home, to see if the problem goes away or moves to
something else.

What is happening is that when ext4 starts writing, it needs memory for the
page cache.  To get that space, the VM system is trying to get space, so it
calls among other things, shrink_icache_memory, to shrink the inode cache,
which then calls clear_inode, which ended up calling ext3_dquot_drop, through
the quota code.  Ext3_dquot_drop() calls journal_start(), which then trips over
a sanity check.   In any case, given what you've told me, I'm even more
convinced this is an ext3 and/or a quota system bug, which is getting triggered
to severe memory pressure caused by ext4 and the fact that you have a very tiny
amount of memory.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688





------- Comment #8 from vedranf@vedranf.mine.nu  2009-01-19 00:50 -------
Errr, that was output of $ free -m, so it's ~1.7GB of memory.
Regarding noatime... I didn't change debian defaults. 
Disabling quotas (/etc/init.d/quota stop) seems to fix the problem. I copied
some 70GB and oops didn't happen.

I need to test it more,  but I'm 90% sure it's quota related bug. You can
assign this bug to someone who is in charge of ext3 and quota.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688





------- Comment #9 from jack@suse.cz  2009-01-20 09:18 -------
Ted, I've looked into this and it is a bug in ext4. It should call
grab_cache_page_write_begin() with AOP_FLAG_NOFS since it cannot handle
recursion into the filesystem for freeing memory. The problem is that at the
time grab_cache_page_write_begin() is called, a transaction is already started
- this is what quota code complains about (because it also needs to start a
transaction but current->journal_info is already occupied by ext4).

If we wanted to avoid AOP_FLAG_NOFS we could get the page, start a transaction,
lock and check the page but that's going to be a bit nasty.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688





------- Comment #10 from jack@suse.cz  2009-02-05 08:30 -------
Created an attachment (id=20124)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=20124&action=view)
Fix for the bug in ext4_write_begin and ext4_da_write_begin

This patch should fix the problem. Testing welcome ;)


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 11688] Oops while writing on a newly ext4 formated 1TB partition on a external USB drive

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=11688


jack@suse.cz changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|fs_ext3@kernel-bugs.osdl.org|jack@suse.cz
             Status|NEW                         |ASSIGNED




------- Comment #11 from jack@suse.cz  2009-02-24 08:22 -------
OK, the patch was merged into ext4-tree. Closing the bug.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
You are the assignee for the bug, or are watching the assignee.