linux-ext4.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* EXT4 panic at jbd2_journal_put_journal_head() in 3.9+
@ 2013-05-09  7:59 EUNBONG SONG
  2013-05-09 15:31 ` Theodore Ts'o
  0 siblings, 1 reply; 9+ messages in thread
From: EUNBONG SONG @ 2013-05-09  7:59 UTC (permalink / raw)
  To: tytso, linux-ext4, linux-kernel@vger.kernel.org


Hello.
In my board, i ran the iozone with multi-thread option. 
My board has 8 cores and i enabled CONFIG_SMP. 
the iozone command as follow: iozone -l 20 -u 20 -r 64k -s 5m -o -F /user/f1 /user/f2 /user/f3 /user/f4 /user/f5 /user/f6 /user/f7 /user/f8 /user/f9 /user/f10 /user/f11 /user/f12 /user/f13 /user/f14 /user/f15 /user/f16 /user/f17 /user/f18 /user/f19 /user/f20

mount info as follow
cat /proc/mounts
/dev/sda3 /user ext4 rw,noatime,nodiratime,errors=panic,barrier=1,data=ordered 0 0

I got a message as below every time i ran iozone test. 


[ 4876.293124] [<ffffffff80272d18>] show_stack+0x68/0x80
[ 4876.309411] [<ffffffff802bde4c>] notifier_call_chain+0x5c/0xa8
[ 4876.315245] [<ffffffff802be524>] __atomic_notifier_call_chain+0x3c/0x58
[ 4876.321860] [<ffffffff802be588>] notify_die+0x38/0x48
[ 4876.326913] [<ffffffff80272548>] do_trap_or_bp+0x48/0x1a8
[ 4876.332312] [<ffffffff8026cfe4>] resume_userspace_check+0x0/0x10
[ 4876.338323] [<ffffffff80462994>] jbd2_journal_put_journal_head+0xcc/0x250
[ 4876.345111] [<ffffffff804603b4>] __jbd2_journal_remove_checkpoint+0x54/0x130
[ 4876.352160] [<ffffffff8045e630>] jbd2_journal_commit_transaction+0x1318/0x1ad0
[ 4876.359383] [<ffffffff80463f4c>] kjournald2+0x114/0x450
[ 4876.364611] [<ffffffff802b8160>] kthread+0xb8/0xc0
[ 4876.369402] [<ffffffff8026d060>] ret_from_kernel_thread+0x10/0x18

When i ran without multi-thread option, the problem was not ocurred.


Thanks. 


^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: Re: EXT4 panic at jbd2_journal_put_journal_head() in 3.9+
@ 2013-05-10  0:51 EUNBONG SONG
  2013-05-10 17:27 ` Tony Luck
  2013-05-10 19:27 ` Theodore Ts'o
  0 siblings, 2 replies; 9+ messages in thread
From: EUNBONG SONG @ 2013-05-10  0:51 UTC (permalink / raw)
  To: Theodore Ts'o
  Cc: linux-ext4@vger.kernel.org, linux-kernel@vger.kernel.org


> Can you give us the full crash message, (i.e., the panic, the BUG,
> WARN, the registers, etc.), and not the stack trace?

>                       - Ted

Hi, Ted
Actually i try to find the crash point. And i confirmed crash point is in __journal_remove_journal_head() function.
I added some debug code and i found  J_ASSERT_JH is failed for jh->b_transaction. 
My source tree has some modifications only for MIPS architecture. I don't think it does not affect to ext4 operation. 
Also I confirmed the problem is not reproduced before merge 149b306089b88e186942a8d6647028ae6683aaf9.

149b306089b88e186942a8d6647028ae6683aaf9 Merge tag 'ext4_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4

My full crash messages are as follows.

Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-powerpc64

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Roo[  233.458766] CPU: 3 PID: 1535 Comm: iozone Not tainted 3.9.0+ #81
t,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darre[  233.470132] Stack :n Sawyer.
                     Ben England.

        Run began: Sun Dec 10  ffffffff8101143808:38:24 2000

        Record Size 64 KB
        File size set to 5120 KB
 a80000008b64b000       SYNC Mode.
        Command line used: iozone -l 20 -u 20 -r 64k -s 5 0000000000000003m -o -F /user/f1 /user/f2 /user/f3 /user/f4 /user/f5 /user/f6 /u ffffffff80292470ser/f7 /user/f8 /user/f9 /user/f10 /user/f11 /user/f12 /user/f13
          /user/f14 /user/f15 /user/f16 /user/f17 /user/f18 /user/f19 /us 0000000000000000er/f20
        Output is in Kbytes/sec
        Time Resolution = 0.000001 se ffffffff80fa0000conds.
        Processor cache size set to 1024 Kbytes.
        Processor ca 000000000000001fche line size set to 32 bytes.
        File stride size set to 17 * re ffffffff80293728cord size.
        Min process = 20
        Max process = 20
        Throughput
         test with 20 processes
        Each process writes a 5120 Kbyte file i 0000000000000000n 64 Kbyte records
 0000000000000000 ffffffff81080000 ffffffff81080000
          ffffffff80e2abf0 ffffffff80f8f9f7 a8000002017a0db8 0000000000000020
          0000000000000003 0000000000000020 a80000020025f968 ffffffff810f0000
          a80000020025f770 ffffffff806ee88c a80000020025f588 ffffffff80290994
          000000007ef10087 ffffffff80293b58 000000000000000a ffffffff80e2abf0
          0000000000000003 a80000020025f4b0 00000002017a10f8 ffffffff805e68b4
          0000000000000000 0000000000000000 0000000000000000 0000000000000000
          0000000000000000 ffffffff80272418 0000000000000000 0000000000000000
          ...
[  233.604041] Call Trace:
[  233.606495] [<ffffffff80272418>] show_stack+0x68/0x80
[  233.611550] [<ffffffff805e68b4>] cdr_event_handler+0x604/0xbf8
[  233.617384] [<ffffffff805e7648>] cdr_event_die+0xd0/0x150
[  233.622784] [<ffffffff802bd42c>] notifier_call_chain+0x5c/0xa8
[  233.628619] [<ffffffff802bdb04>] __atomic_notifier_call_chain+0x3c/0x58
[  233.635233] [<ffffffff802bdb68>] notify_die+0x38/0x48
[  233.640285] [<ffffffff80271c48>] do_trap_or_bp+0x48/0x1a8
[  233.645684] [<ffffffff8026c764>] resume_userspace_check+0x0/0x10
[  233.651695] [<ffffffff80460b64>] jbd2_journal_put_journal_head+0xcc/0x250
[  233.658484] [<ffffffff8045a1b4>] jbd2_journal_get_write_access+0x3c/0x58
[  233.665188] [<ffffffff804348a8>] __ext4_journal_get_write_access+0x58/0xa0
[  233.672064] [<ffffffff80410344>] ext4_reserve_inode_write+0x84/0xb0
[  233.678331] [<ffffffff804103ac>] ext4_mark_inode_dirty+0x3c/0x1e0
[  233.684424] [<ffffffff80410590>] ext4_dirty_inode+0x40/0x70
[  233.689998] [<ffffffff80392258>] __mark_inode_dirty+0x48/0x238
[  233.695832] [<ffffffff803828f4>] update_time+0xb4/0x100
[  233.701058] [<ffffffff803829f0>] file_update_time+0xb0/0x108
[  233.706718] [<ffffffff8031eb98>] __generic_file_aio_write+0x180/0x380
[  233.713158] [<ffffffff8031edf8>] generic_file_aio_write+0x60/0xc0
[  233.719252] [<ffffffff8040af54>] ext4_file_write+0x6c/0x468
[  233.724827] [<ffffffff80366cbc>] do_sync_write+0x84/0xe8
[  233.730139] [<ffffffff80368700>] vfs_write+0xe0/0x1e0
[  233.735191] [<ffffffff803688f8>] SyS_write+0x50/0xc0
[  233.740157] [<ffffffff80274864>] handle_sys64+0x44/0x64

Thanks. 

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: Re: Re: EXT4 panic at jbd2_journal_put_journal_head() in 3.9+
@ 2013-05-13  2:21 EUNBONG SONG
  2013-05-13  3:11 ` Tony Luck
  0 siblings, 1 reply; 9+ messages in thread
From: EUNBONG SONG @ 2013-05-13  2:21 UTC (permalink / raw)
  To: Tony Luck, Dmitry Monakhov
  Cc: Theodore Ts'o, linux-ext4@vger.kernel.org,
	linux-kernel@vger.kernel.org



> CONFIG_IA64_PAGE_SIZE_64KB=y

> fsblock size is whatever is the default for SLES11SP2 on ia64 - which
> tool will tell me?

> My git bisect finally competed and points the a finger at:

> bisect> git bisect good
> ae4647fb7654676fc44a97e86eb35f9f06b99f66 is first bad commit
> commit ae4647fb7654676fc44a97e86eb35f9f06b99f66
> Author: Jan Kara 
> Date:   Fri Apr 12 00:03:42 2013 -0400

>     jbd2: reduce journal_head size

>     Remove unused t_cow_tid field (ext4 copy-on-write support doesn't seem
>     to be happening) and change b_modified and b_jlist to bitfields thus
>     saving 8 bytes in the structure.

>     Signed-off-by: Jan Kara 
>     Signed-off-by: "Theodore Ts'o" 
>     Reviewed-by: Zheng Liu 

> :040000 040000 c39ece4341894b3daf84764ba425a87ffb90fe50
> d4e8d9185c2a1b740c235ca8ed05d496a442fce3 M      include

Hi, my git bisect result is same yours. And i reported that to community yesterday.
Thanks. 

^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-05-13  6:14 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-05-09  7:59 EXT4 panic at jbd2_journal_put_journal_head() in 3.9+ EUNBONG SONG
2013-05-09 15:31 ` Theodore Ts'o
  -- strict thread matches above, loose matches on Subject: below --
2013-05-10  0:51 EUNBONG SONG
2013-05-10 17:27 ` Tony Luck
2013-05-11  7:52   ` Dmitry Monakhov
2013-05-13  2:04     ` Tony Luck
2013-05-13  3:07       ` Theodore Ts'o
2013-05-13  5:06         ` Sidorov, Andrei
2013-05-10 19:27 ` Theodore Ts'o
2013-05-10 20:38   ` David Daney
2013-05-13  2:21 Re: " EUNBONG SONG
2013-05-13  3:11 ` Tony Luck
2013-05-13  3:36   ` Theodore Ts'o
2013-05-13  5:18     ` Mike Galbraith
2013-05-13  6:14       ` Tony Luck

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).