2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

Hi

I got this with the latest git (2150edc6c5cf00f7adb54538b9ea2a3e9cedca3f).

Assertion failed: fs_is_ok, file: fs/xfs/xfs_btree.c, line: 3327
------------[ cut here ]------------
kernel BUG at fs/xfs/support/debug.c:108!
invalid opcode: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
last sysfs file: /sys/devices/platform/w83627hf.656/name
Modules linked in: w83627hf hwmon_vid i2c_nforce2

Pid: 250, comm: pdflush Not tainted (2.6.28-07966-g2150edc #1)
EIP: 0060:[<c029cfce>] EFLAGS: 00010282 CPU: 0
EIP is at assfail+0x1e/0x30
EAX: 00000053 EBX: ef2ed170 ECX: 10000000 EDX: 10000000
ESI: 00000000 EDI: f6b158a4 EBP: f6b15838 ESP: f6b15828
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process pdflush (pid: 250, ti=f6b14000 task=f6b0d1c0 task.ti=f6b14000)
Stack:
 c04bbeb0 c049ba44 c049c4d3 00000cff f6b158f0 c024f3ed 00000008 f0ba7000
 f6b15894 f6b15878 f6b158e0 f0ba7000 00000010 f6b158fc 00000000 ef2ed170
 00000000 f0b98000 f0ba7000 00000000 0000000b c024bb8d f6b158e8 f6b15890
Call Trace:
 [<c024f3ed>] ? xfs_btree_delrec+0xebd/0x1270
 [<c024bb8d>] ? xfs_btree_lookup_get_block+0x9d/0x100
 [<c02492bd>] ? xfs_bmbt_init_key_from_rec+0xd/0x20
 [<c024a260>] ? xfs_lookup_get_search_key+0x40/0x80
 [<c024f7cc>] ? xfs_btree_delete+0x2c/0xa0
 [<c02431bb>] ? xfs_bmap_add_extent_delay_real+0x14bb/0x16f0
 [<c01838fc>] ? slab_pad_check+0x3c/0x120
 [<c01832ed>] ? check_object+0x13d/0x200
 [<c02440c6>] ? xfs_bmap_add_extent+0x626/0x670
 [<c02490ec>] ? xfs_bmbt_init_cursor+0x2c/0x100
 [<c0247ae8>] ? xfs_bmapi+0xfc8/0x1c80
 [<c014d076>] ? __lock_acquire+0x2b6/0x1190
 [<c014d076>] ? __lock_acquire+0x2b6/0x1190
 [<c02714c4>] ? xfs_iomap_write_allocate+0x254/0x450
 [<c0275cd0>] ? xfs_log_move_tail+0x190/0x1d0
 [<c02725f7>] ? xfs_iomap+0x3a7/0x3f0
 [<c014d076>] ? __lock_acquire+0x2b6/0x1190
 [<c02916fd>] ? xfs_page_state_convert+0x32d/0x7b0
 [<c014c8bc>] ? mark_held_locks+0x4c/0x90
 [<c0291cae>] ? xfs_vm_writepage+0x5e/0xf0
 [<c0165c3b>] ? __writepage+0xb/0x40
 [<c0166ceb>] ? write_cache_pages+0x1ab/0x370
 [<c0165c30>] ? __writepage+0x0/0x40
 [<c0166ed3>] ? generic_writepages+0x23/0x30
 [<c028faa1>] ? xfs_vm_writepages+0x41/0x50
 [<c028fa60>] ? xfs_vm_writepages+0x0/0x50
 [<c0166f0e>] ? do_writepages+0x2e/0x50
 [<c01a3342>] ? __writeback_single_inode+0x82/0x340
 [<c01a3756>] ? generic_sync_sb_inodes+0x26/0x390
 [<c04142d6>] ? _spin_lock+0x66/0x70
 [<c01a3a22>] ? generic_sync_sb_inodes+0x2f2/0x390
 [<c01a3c66>] ? writeback_inodes+0x56/0xe0
 [<c0165f0b>] ? wb_kupdate+0x7b/0xf0
 [<c0167530>] ? pdflush+0x0/0x190
 [<c0167600>] ? pdflush+0xd0/0x190
 [<c0165e90>] ? wb_kupdate+0x0/0xf0
 [<c013be0a>] ? kthread+0x3a/0x70
 [<c013bdd0>] ? kthread+0x0/0x70
 [<c0103b83>] ? kernel_thread_helper+0x7/0x14
Code: 00 e8 17 87 02 00 c9 c3 90 8d 74 26 00 55 89 e5 83 ec 10 89 4c 24 0c 89 54 24 08 89 44 24 04 c7 04 24 b0 be 4b c0 e8 fb 48 17 00 <0f> 0b eb fe 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 89 e5
EIP: [<c029cfce>] assfail+0x1e/0x30 SS:ESP 0068:f6b15828

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Dave Chinner]

On Fri, Jan 09, 2009 at 07:41:21AM +0300, Alexander Beregalov wrote:
> Hi
> 
> I got this with the latest git (2150edc6c5cf00f7adb54538b9ea2a3e9cedca3f).
> 
> Assertion failed: fs_is_ok, file: fs/xfs/xfs_btree.c, line: 3327
> ------------[ cut here ]------------
> kernel BUG at fs/xfs/support/debug.c:108!
> invalid opcode: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
> last sysfs file: /sys/devices/platform/w83627hf.656/name
> Modules linked in: w83627hf hwmon_vid i2c_nforce2
> 
> Pid: 250, comm: pdflush Not tainted (2.6.28-07966-g2150edc #1)
> EIP: 0060:[<c029cfce>] EFLAGS: 00010282 CPU: 0
> EIP is at assfail+0x1e/0x30
> EAX: 00000053 EBX: ef2ed170 ECX: 10000000 EDX: 10000000
> ESI: 00000000 EDI: f6b158a4 EBP: f6b15838 ESP: f6b15828
>  DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
> Process pdflush (pid: 250, ti=f6b14000 task=f6b0d1c0 task.ti=f6b14000)
> Stack:
>  c04bbeb0 c049ba44 c049c4d3 00000cff f6b158f0 c024f3ed 00000008 f0ba7000
>  f6b15894 f6b15878 f6b158e0 f0ba7000 00000010 f6b158fc 00000000 ef2ed170
>  00000000 f0b98000 f0ba7000 00000000 0000000b c024bb8d f6b158e8 f6b15890
> Call Trace:
>  [<c024f3ed>] ? xfs_btree_delrec+0xebd/0x1270
>  [<c024bb8d>] ? xfs_btree_lookup_get_block+0x9d/0x100
>  [<c02492bd>] ? xfs_bmbt_init_key_from_rec+0xd/0x20
>  [<c024a260>] ? xfs_lookup_get_search_key+0x40/0x80
>  [<c024f7cc>] ? xfs_btree_delete+0x2c/0xa0

Looks like the same btree corruption problem as this:

http://oss.sgi.com/archives/xfs/2009-01/msg00118.html

We're working to find the cause right now, but I find it interesting
that you are running a CONFIG_XFS_DEBUG=y kernel and the first
point that something is found wrong is the same place that the
above non-debug case is detecting corruption.

Can you reproduce it/have a reproducable test case?

Cheers,

Dave.
-- 
Dave Chinner
david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

2009/1/9 Dave Chinner <david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org>:
> On Fri, Jan 09, 2009 at 07:41:21AM +0300, Alexander Beregalov wrote:
> Looks like the same btree corruption problem as this:
>
> http://oss.sgi.com/archives/xfs/2009-01/msg00118.html
>
> We're working to find the cause right now, but I find it interesting
> that you are running a CONFIG_XFS_DEBUG=y kernel and the first
> point that something is found wrong is the same place that the
> above non-debug case is detecting corruption.
>
> Can you reproduce it/have a reproducable test case?
Yes, it is easy reproducible for me, it happens every time I run
`rtorrent` during few minutes after boot.
I will try to provide simpler testcase.

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

2009/1/10 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> 2009/1/9 Dave Chinner <david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org>:
>> On Fri, Jan 09, 2009 at 07:41:21AM +0300, Alexander Beregalov wrote:
>> Looks like the same btree corruption problem as this:
>>
>> http://oss.sgi.com/archives/xfs/2009-01/msg00118.html
>>
>> We're working to find the cause right now, but I find it interesting
>> that you are running a CONFIG_XFS_DEBUG=y kernel and the first
>> point that something is found wrong is the same place that the
>> above non-debug case is detecting corruption.
>>
>> Can you reproduce it/have a reproducable test case?
> Yes, it is easy reproducible for me, it happens every time I run
> `rtorrent` during few minutes after boot.
> I will try to provide simpler testcase.
>

This time I get it during log replaing before mounting.

* Mounting local filesystems...
[  214.039447] XFS mounting filesystem sda3
[  214.190773] Ending clean XFS mount for filesystem: sda3
[  214.321525] XFS mounting filesystem sdb1
[  214.453029] Starting XFS recovery on filesystem: sdb1 (logdev: internal)
[  215.062964] Ending XFS recovery on filesystem: sdb1 (logdev: internal)
[  215.163358] Assertion failed: fs_is_ok, file: fs/xfs/xfs_btree.c, line: 3327
[  215.248223] ------------[ cut here ]------------
[  215.257815] kernel BUG at fs/xfs/support/debug.c:108!
[  215.257815] invalid opcode: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
[  215.257815] last sysfs file: /sys/kernel/uevent_seqnum
[  215.257815] Modules linked in: i2c_nforce2
[  215.257815]
[  215.257815] Pid: 764, comm: async/1 Not tainted (2.6.28-08160-g7c51d57 #1)

It seems I need to downgrade the kernel to boot the machine.

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

2009/1/10 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> 2009/1/10 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>> 2009/1/9 Dave Chinner <david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org>:
>>> On Fri, Jan 09, 2009 at 07:41:21AM +0300, Alexander Beregalov wrote:
>>> Looks like the same btree corruption problem as this:
>>>
>>> http://oss.sgi.com/archives/xfs/2009-01/msg00118.html
>>>
>>> We're working to find the cause right now, but I find it interesting
>>> that you are running a CONFIG_XFS_DEBUG=y kernel and the first
>>> point that something is found wrong is the same place that the
>>> above non-debug case is detecting corruption.
>>>
>>> Can you reproduce it/have a reproducable test case?
>> Yes, it is easy reproducible for me, it happens every time I run
>> `rtorrent` during few minutes after boot.
>> I will try to provide simpler testcase.
>>
>
> This time I get it during log replaing before mounting.
>
> * Mounting local filesystems...
> [  214.039447] XFS mounting filesystem sda3
> [  214.190773] Ending clean XFS mount for filesystem: sda3
> [  214.321525] XFS mounting filesystem sdb1
> [  214.453029] Starting XFS recovery on filesystem: sdb1 (logdev: internal)
> [  215.062964] Ending XFS recovery on filesystem: sdb1 (logdev: internal)
> [  215.163358] Assertion failed: fs_is_ok, file: fs/xfs/xfs_btree.c, line: 3327

Dave, how can I extract useful information from the log? How can I
replay it by one transaction?
Perhaps it is possibly to find  the trasaction which triggers the bug.
I used `xfs_logprint -C` to save the log, but it is more than 100Mb.

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

2009/1/10 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> 2009/1/10 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>> 2009/1/10 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>>> 2009/1/9 Dave Chinner <david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org>:
>>>> On Fri, Jan 09, 2009 at 07:41:21AM +0300, Alexander Beregalov wrote:
>>>> Looks like the same btree corruption problem as this:
>>>>
>>>> http://oss.sgi.com/archives/xfs/2009-01/msg00118.html
>>>>
>>>> We're working to find the cause right now, but I find it interesting
>>>> that you are running a CONFIG_XFS_DEBUG=y kernel and the first
>>>> point that something is found wrong is the same place that the
>>>> above non-debug case is detecting corruption.
>>>>
>>>> Can you reproduce it/have a reproducable test case?
>>> Yes, it is easy reproducible for me, it happens every time I run
>>> `rtorrent` during few minutes after boot.
>>> I will try to provide simpler testcase.

It is hard to bisect it,
These are last good and bad commits:
good: [854929f05831d3a290a802815ee955b96c740c61] [XFS] add new btree statistics
bad: [7f7c39ccb6045cf1fd5e7684a484c445291b44d4] [XFS] make btree tracing generic

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Christoph Hellwig]

On Sat, Jan 10, 2009 at 03:19:00PM +0300, Alexander Beregalov wrote:
> It is hard to bisect it,
> These are last good and bad commits:
> good: [854929f05831d3a290a802815ee955b96c740c61] [XFS] add new btree statistics
> bad: [7f7c39ccb6045cf1fd5e7684a484c445291b44d4] [XFS] make btree tracing generic

That would be odd as 7f7c39ccb6045cf1fd5e7684a484c445291b44d4 only
changes the tracing code which currently isn't enabled.  Or we
get some sort of miscompilation due slightly different noop
macros.

How big is the filesystem where you see this corruption? Maybe I could
reproduce it locally with a xfs_metadump image.

> 
> _______________________________________________
> xfs mailing list
> xfs-VZNHf3L845pBDgjK7y7TUQ@public.gmane.org
> http://oss.sgi.com/mailman/listinfo/xfs
---end quoted text---

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

2009/1/10 Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>:
> On Sat, Jan 10, 2009 at 03:19:00PM +0300, Alexander Beregalov wrote:
>> It is hard to bisect it,
>> These are last good and bad commits:
>> good: [854929f05831d3a290a802815ee955b96c740c61] [XFS] add new btree statistics
>> bad: [7f7c39ccb6045cf1fd5e7684a484c445291b44d4] [XFS] make btree tracing generic
>
> That would be odd as 7f7c39ccb6045cf1fd5e7684a484c445291b44d4 only
> changes the tracing code which currently isn't enabled.  Or we
> get some sort of miscompilation due slightly different noop
> macros.
I meant the first bad commit is between these two commits. All of them
fail to compile as is,
I added xfs_btree_trace.h manually to compile it, I got different bugs
on these commits,
but I am not sure if they are really different. Like this:

Filesystem "sdb1": XFS internal error xfs_btree_check_lblock at line
84 of file fs/xfs/xfs_btree.c.  Caller 0xc0247322
Pid: 247, comm: pdflush Not tainted 2.6.28-rc2-00219-g3cc7524 #3
Call Trace:
 [<c025d8d2>] ? xfs_cmn_err+0x32/0x60
 [<c025d94e>] xfs_error_report+0x4e/0x50
 [<c0247322>] ? xfs_btree_check_block+0x32/0x40
 [<c02471ad>] xfs_btree_check_lblock+0x4d/0x190
 [<c0247322>] ? xfs_btree_check_block+0x32/0x40
 [<c0283d5d>] ? xfs_trans_read_buf+0x46d/0x530
 [<c0247322>] xfs_btree_check_block+0x32/0x40
 [<c0247504>] xfs_btree_read_buf_block+0xe4/0x100
 [<c024979f>] xfs_btree_rshift+0xcf/0x540
 [<c025d9eb>] ? xfs_error_test+0x1b/0xc0
 [<c025d9eb>] ? xfs_error_test+0x1b/0xc0
 [<c024b652>] xfs_btree_make_block_unfull+0x32/0x140
 [<c0244d62>] ? xfs_bmbt_recs_inorder+0x32/0x70
 [<c024bd9e>] xfs_btree_insrec+0x63e/0x6c0
 [<c024be89>] xfs_btree_insert+0x69/0x190
 [<c023eaad>] xfs_bmap_add_extent_delay_real+0x142d/0x1700
 [<c0180cfc>] ? slab_pad_check+0x3c/0x120
 [<c022dc79>] ? xfs_alloc_vextent+0x2e9/0x760
 [<c01806bd>] ? check_object+0x13d/0x200
 [<c023fa56>] xfs_bmap_add_extent+0x626/0x670
 [<c0244b2c>] ? xfs_bmbt_init_cursor+0x2c/0x100
 [<c024349b>] xfs_bmapi+0xfcb/0x1c90
 [<c014b7a5>] ? __lock_acquire+0x2c5/0x1000
 [<c014b7a5>] ? __lock_acquire+0x2c5/0x1000
 [<c014b7a5>] ? __lock_acquire+0x2c5/0x1000
 [<c026d6e4>] xfs_iomap_write_allocate+0x254/0x450
 [<c0272010>] ? xfs_log_move_tail+0x190/0x1d0
 [<c0272010>] ? xfs_log_move_tail+0x190/0x1d0
 [<c026e8e7>] xfs_iomap+0x3a7/0x3f0
 [<c014b1cc>] ? trace_hardirqs_on_caller+0x7c/0x160
 [<c028d68e>] xfs_map_blocks+0x3e/0x90
 [<c028f1ea>] xfs_page_state_convert+0x2ea/0x740
 [<c012cb32>] ? _local_bh_enable+0x52/0xa0

Anyway, If they are different, I can not catch the bug at xfs_btree_delrec()
beacuse the bug xfs_btree.c:84 happens earlierly.

>
> How big is the filesystem where you see this corruption? Maybe I could
> reproduce it locally with a xfs_metadump image.

They are big enough, I have four such FS's for >200Gb

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Christoph Hellwig]

On Sat, Jan 10, 2009 at 06:09:05PM +0300, Alexander Beregalov wrote:
> > That would be odd as 7f7c39ccb6045cf1fd5e7684a484c445291b44d4 only
> > changes the tracing code which currently isn't enabled.  Or we
> > get some sort of miscompilation due slightly different noop
> > macros.
> I meant the first bad commit is between these two commits. All of them
> fail to compile as is,
> I added xfs_btree_trace.h manually to compile it, I got different bugs
> on these commits,
> but I am not sure if they are really different. Like this:

Ah crap.  When lachlan checked in the btree tracing he forgot to
add that header and it only got in after that.  Can you bisect
further between those commit by just using xfs_btree_trce.h from
a newer version?  It hasn't had a single change yet since it was
commited.

This is quite important as all changes between these two revisions
are quite large and deal with consolidating the btree code.

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

On Sat, Jan 10, 2009 at 10:28:03AM -0500, Christoph Hellwig wrote:
> On Sat, Jan 10, 2009 at 06:09:05PM +0300, Alexander Beregalov wrote:
> > > That would be odd as 7f7c39ccb6045cf1fd5e7684a484c445291b44d4 only
> > > changes the tracing code which currently isn't enabled.  Or we
> > > get some sort of miscompilation due slightly different noop
> > > macros.
> > I meant the first bad commit is between these two commits. All of them
> > fail to compile as is,
> > I added xfs_btree_trace.h manually to compile it, I got different bugs
> > on these commits,
> > but I am not sure if they are really different. Like this:
> 
> Ah crap.  When lachlan checked in the btree tracing he forgot to
> add that header and it only got in after that.  Can you bisect
> further between those commit by just using xfs_btree_trce.h from
> a newer version?  It hasn't had a single change yet since it was
> commited.
> 
> This is quite important as all changes between these two revisions
> are quite large and deal with consolidating the btree code.
> 

I can not reproduce it now, I get the following message instead:
I will try to repair the filesystem.

Filesystem "sdb1": XFS internal error xfs_btree_check_lblock at line 86 of file fs/xfs/xfs_btree.c.  Caller 0xc024af42

Pid: 251, comm: pdflush Not tainted 2.6.28-09244-g3d14bda #4
Call Trace:
 [<c0261462>] ? xfs_cmn_err+0x32/0x60
 [<c02614de>] xfs_error_report+0x4e/0x50
 [<c024af42>] ? xfs_btree_check_block+0x32/0x40
 [<c024adcd>] xfs_btree_check_lblock+0x4d/0x190
 [<c024af42>] ? xfs_btree_check_block+0x32/0x40
 [<c0286bf0>] ? xfs_trans_read_buf+0x470/0x530
 [<c024af42>] xfs_btree_check_block+0x32/0x40
 [<c024b124>] xfs_btree_read_buf_block+0xe4/0x100
 [<c024ce2d>] xfs_btree_lshift+0xbd/0x580
 [<c026157b>] ? xfs_error_test+0x1b/0xc0
 [<c024f29b>] xfs_btree_make_block_unfull+0x5b/0x140
 [<c0248972>] ? xfs_bmbt_recs_inorder+0x32/0x70
 [<c024f9be>] xfs_btree_insrec+0x63e/0x6c0
 [<c024faa9>] xfs_btree_insert+0x69/0x190
 [<c024276b>] xfs_bmap_add_extent_delay_real+0x141b/0x16f0
 [<c0181dcc>] ? slab_pad_check+0x3c/0x120
 [<c02319f4>] ? xfs_alloc_vextent+0x2d4/0x730
 [<c018305d>] ? check_object+0x13d/0x200
 [<c0243716>] xfs_bmap_add_extent+0x626/0x670
 [<c024873c>] ? xfs_bmbt_init_cursor+0x2c/0x100
 [<c0247138>] xfs_bmapi+0xfc8/0x1c80
 [<c014ccd6>] ? __lock_acquire+0x2b6/0x1190
 [<c014ccd6>] ? __lock_acquire+0x2b6/0x1190
 [<c0270b24>] xfs_iomap_write_allocate+0x254/0x450
 [<c0275350>] ? xfs_log_move_tail+0x190/0x1d0
 [<c0271c57>] xfs_iomap+0x3a7/0x3f0
 [<c014ccd6>] ? __lock_acquire+0x2b6/0x1190
 [<c0290d3d>] xfs_page_state_convert+0x3fd/0x790
 [<c014c51c>] ? mark_held_locks+0x4c/0x90
 [<c02911fa>] xfs_vm_writepage+0x5a/0xf0
 [<c016580b>] __writepage+0xb/0x40
 [<c01662ab>] write_cache_pages+0x19b/0x350
 [<c0165800>] ? __writepage+0x0/0x40
 [<c0166483>] generic_writepages+0x23/0x30
 [<c028f2d1>] xfs_vm_writepages+0x41/0x50
 [<c028f290>] ? xfs_vm_writepages+0x0/0x50
 [<c01664be>] do_writepages+0x2e/0x50
 [<c01a2b02>] __writeback_single_inode+0x82/0x340
 [<c01a2f16>] ? generic_sync_sb_inodes+0x26/0x390
 [<c0413246>] ? _spin_lock+0x66/0x70
 [<c01a31e2>] generic_sync_sb_inodes+0x2f2/0x390
 [<c01a3426>] writeback_inodes+0x56/0xe0
 [<c01665fb>] wb_kupdate+0x7b/0xf0
 [<c0167060>] ? pdflush+0x0/0x190
 [<c0167130>] pdflush+0xd0/0x190
 [<c0166580>] ? wb_kupdate+0x0/0xf0
 [<c013ba2a>] kthread+0x3a/0x70
 [<c013b9f0>] ? kthread+0x0/0x70
 [<c0103b83>] kernel_thread_helper+0x7/0x14
Filesystem "sdb1": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c.  Caller 0xc0270c1c

Pid: 251, comm: pdflush Not tainted 2.6.28-09244-g3d14bda #4
Call Trace:
 [<c0261462>] ? xfs_cmn_err+0x32/0x60
 [<c02614de>] xfs_error_report+0x4e/0x50
 [<c0270c1c>] ? xfs_iomap_write_allocate+0x34c/0x450
 [<c02843e6>] xfs_trans_cancel+0x106/0x1e0
 [<c0270c1c>] ? xfs_iomap_write_allocate+0x34c/0x450
 [<c0270c1c>] xfs_iomap_write_allocate+0x34c/0x450
 [<c0275350>] ? xfs_log_move_tail+0x190/0x1d0
 [<c0271c57>] xfs_iomap+0x3a7/0x3f0
 [<c014ccd6>] ? __lock_acquire+0x2b6/0x1190
 [<c0290d3d>] xfs_page_state_convert+0x3fd/0x790
 [<c014c51c>] ? mark_held_locks+0x4c/0x90
 [<c02911fa>] xfs_vm_writepage+0x5a/0xf0
 [<c016580b>] __writepage+0xb/0x40
 [<c01662ab>] write_cache_pages+0x19b/0x350
 [<c0165800>] ? __writepage+0x0/0x40
 [<c0166483>] generic_writepages+0x23/0x30
 [<c028f2d1>] xfs_vm_writepages+0x41/0x50
 [<c028f290>] ? xfs_vm_writepages+0x0/0x50
 [<c01664be>] do_writepages+0x2e/0x50
 [<c01a2b02>] __writeback_single_inode+0x82/0x340
 [<c01a2f16>] ? generic_sync_sb_inodes+0x26/0x390
 [<c0413246>] ? _spin_lock+0x66/0x70
 [<c01a31e2>] generic_sync_sb_inodes+0x2f2/0x390
 [<c01a3426>] writeback_inodes+0x56/0xe0
 [<c01665fb>] wb_kupdate+0x7b/0xf0
 [<c0167060>] ? pdflush+0x0/0x190
 [<c0167130>] pdflush+0xd0/0x190
 [<c0166580>] ? wb_kupdate+0x0/0xf0
 [<c013ba2a>] kthread+0x3a/0x70
 [<c013b9f0>] ? kthread+0x0/0x70
 [<c0103b83>] kernel_thread_helper+0x7/0x14
xfs_force_shutdown(sdb1,0x8) called from line 1165 of file fs/xfs/xfs_trans.c.  Return address = 0xc02843ff
XFS: Transforming an alert into a BUG.
Filesystem "sdb1": Corruption of in-memory data detected.  Shutting down filesystem: sdb1
------------[ cut here ]------------
kernel BUG at fs/xfs/support/debug.c:101!
invalid opcode: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
last sysfs file: /sys/devices/platform/w83627hf.656/name
Modules linked in: w83627hf hwmon_vid i2c_nforce2

Pid: 251, comm: pdflush Not tainted (2.6.28-09244-g3d14bda #4)  
EIP: 0060:[<c029c559>] EFLAGS: 00010246 CPU: 0
EIP is at xfs_fs_vcmn_err+0xc9/0xd0
EAX: f6b06000 EBX: f73e3aa0 ECX: 10000000 EDX: 10000000
ESI: 00000000 EDI: c04b9fc8 EBP: f6b07b74 ESP: f6b07b58
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process pdflush (pid: 251, ti=f6b06000 task=f6b0bd50 task.ti=f6b06000)
Stack:
 c04c2605 c0488fe5 c0aeafc0 00000292 00000008 00000000 f73e3aa0 f6b07b84
 c0261462 f6b07b9c c04b4c58 f6b07bb8 c028e2c2 00000010 00000000 f73e3aa0
 c04b9fc8 f7345cb0 c049d105 c02843ff 00000000 00000008 f6b07c70 00000001
Call Trace:
 [<c0261462>] ? xfs_cmn_err+0x32/0x60
 [<c028e2c2>] ? xfs_do_force_shutdown+0x152/0x1e0
 [<c02843ff>] ? xfs_trans_cancel+0x11f/0x1e0
 [<c02843ff>] ? xfs_trans_cancel+0x11f/0x1e0
 [<c0270c1c>] ? xfs_iomap_write_allocate+0x34c/0x450
 [<c0270c1c>] ? xfs_iomap_write_allocate+0x34c/0x450
 [<c0275350>] ? xfs_log_move_tail+0x190/0x1d0
 [<c0271c57>] ? xfs_iomap+0x3a7/0x3f0
 [<c014ccd6>] ? __lock_acquire+0x2b6/0x1190
 [<c0290d3d>] ? xfs_page_state_convert+0x3fd/0x790
 [<c014c51c>] ? mark_held_locks+0x4c/0x90
 [<c02911fa>] ? xfs_vm_writepage+0x5a/0xf0
 [<c016580b>] ? __writepage+0xb/0x40
 [<c01662ab>] ? write_cache_pages+0x19b/0x350
 [<c0165800>] ? __writepage+0x0/0x40
 [<c0166483>] ? generic_writepages+0x23/0x30
 [<c028f2d1>] ? xfs_vm_writepages+0x41/0x50
 [<c028f290>] ? xfs_vm_writepages+0x0/0x50
 [<c01664be>] ? do_writepages+0x2e/0x50
 [<c01a2b02>] ? __writeback_single_inode+0x82/0x340
 [<c01a2f16>] ? generic_sync_sb_inodes+0x26/0x390
 [<c0413246>] ? _spin_lock+0x66/0x70
 [<c01a31e2>] ? generic_sync_sb_inodes+0x2f2/0x390
 [<c01a3426>] ? writeback_inodes+0x56/0xe0
 [<c01665fb>] ? wb_kupdate+0x7b/0xf0
 [<c0167060>] ? pdflush+0x0/0x190
 [<c0167130>] ? pdflush+0xd0/0x190
 [<c0166580>] ? wb_kupdate+0x0/0xf0
 [<c013ba2a>] ? kthread+0x3a/0x70
 [<c013b9f0>] ? kthread+0x0/0x70
 [<c0103b83>] ? kernel_thread_helper+0x7/0x14
Code: e8 06 43 17 00 8b 55 f0 b8 20 b0 51 c0 e8 f0 75 17 00 85 f6 74 15 83 c4 10 5b 5e 5f c9 c3 8d 74 26 00 c6 81 c0 af ae c0 00 eb bb <0f> 0b eb fe 8d 76 00 55 b8 20 b0 51 c0 89 e5 57 56 53 83 ec 0c 
EIP: [<c029c559>] xfs_fs_vcmn_err+0xc9/0xd0 SS:ESP 0068:f6b07b58
e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
  Tx Queue             <0>
  TDH                  <3f>
  TDT                  <3f>
  next_to_use          <3f>
  next_to_clean        <ca>
buffer_info[next_to_clean]
  time_stamp           <ce70>
  next_to_watch        <ca>
  jiffies              <cfc6>
  next_to_watch.status <1>
---[ end trace 81071308b66cb6c7 ]---

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Dave Chinner]

On Sun, Jan 11, 2009 at 01:14:59AM +0300, Alexander Beregalov wrote:
> On Sat, Jan 10, 2009 at 10:28:03AM -0500, Christoph Hellwig wrote:
> > On Sat, Jan 10, 2009 at 06:09:05PM +0300, Alexander Beregalov wrote:
> > > > That would be odd as 7f7c39ccb6045cf1fd5e7684a484c445291b44d4 only
> > > > changes the tracing code which currently isn't enabled.  Or we
> > > > get some sort of miscompilation due slightly different noop
> > > > macros.
> > > I meant the first bad commit is between these two commits. All of them
> > > fail to compile as is,
> > > I added xfs_btree_trace.h manually to compile it, I got different bugs
> > > on these commits,
> > > but I am not sure if they are really different. Like this:
> > 
> > Ah crap.  When lachlan checked in the btree tracing he forgot to
> > add that header and it only got in after that.  Can you bisect
> > further between those commit by just using xfs_btree_trce.h from
> > a newer version?  It hasn't had a single change yet since it was
> > commited.
> > 
> > This is quite important as all changes between these two revisions
> > are quite large and deal with consolidating the btree code.
> > 
> 
> I can not reproduce it now, I get the following message instead:
> I will try to repair the filesystem.
> 
> Filesystem "sdb1": XFS internal error xfs_btree_check_lblock at line 86 of file fs/xfs/xfs_btree.c.  Caller 0xc024af42
> 
> Pid: 251, comm: pdflush Not tainted 2.6.28-09244-g3d14bda #4
> Call Trace:
>  [<c0261462>] ? xfs_cmn_err+0x32/0x60
>  [<c02614de>] xfs_error_report+0x4e/0x50
>  [<c024af42>] ? xfs_btree_check_block+0x32/0x40
>  [<c024adcd>] xfs_btree_check_lblock+0x4d/0x190
>  [<c024af42>] ? xfs_btree_check_block+0x32/0x40
>  [<c0286bf0>] ? xfs_trans_read_buf+0x470/0x530
>  [<c024af42>] xfs_btree_check_block+0x32/0x40
>  [<c024b124>] xfs_btree_read_buf_block+0xe4/0x100
>  [<c024ce2d>] xfs_btree_lshift+0xbd/0x580
>  [<c026157b>] ? xfs_error_test+0x1b/0xc0
>  [<c024f29b>] xfs_btree_make_block_unfull+0x5b/0x140
>  [<c0248972>] ? xfs_bmbt_recs_inorder+0x32/0x70
>  [<c024f9be>] xfs_btree_insrec+0x63e/0x6c0
>  [<c024faa9>] xfs_btree_insert+0x69/0x190

Hmmmm - this might be getting closer to the source of the bug.
It's being detecting when reading in the buffer to do a left shift
now, not during the delete of a record.

I'd suggest that you treat this as the same failure and continue
the bisect to try to find when no problems show up at all.

Cheers,

Dave.
-- 
Dave Chinner
david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

> Hmmmm - this might be getting closer to the source of the bug.
> It's being detecting when reading in the buffer to do a left shift
> now, not during the delete of a record.
>
> I'd suggest that you treat this as the same failure and continue
> the bisect to try to find when no problems show up at all.

687b890a184fef263ebb773926e1f4aa69240d01 is the first bad commit.
Does it make sense?
It can not be reverted from the current git to be sure it is guilty
due to merge conflicts.

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

2009/1/12 Alexander Beregalov <a.beregalov-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>> Hmmmm - this might be getting closer to the source of the bug.
>> It's being detecting when reading in the buffer to do a left shift
>> now, not during the delete of a record.
>>
>> I'd suggest that you treat this as the same failure and continue
>> the bisect to try to find when no problems show up at all.
>
> 687b890a184fef263ebb773926e1f4aa69240d01 is the first bad commit.
> Does it make sense?
> It can not be reverted from the current git to be sure it is guilty
> due to merge conflicts.
>

No! I got similar bug message on 9eaead5 two hours later.
9eaead5 is previous to 687b890. 9eaead5 might be the first bad commit
or earlier one.
It seems I need to test the kernel for few hours before mark it good.

Filesystem "sdb1": XFS internal error xfs_btree_check_lblock at line
200 of file fs/xfs/xfs_btree.c.  Caller 0xc024c132
Pid: 2481, comm: pdflush Not tainted 2.6.28-rc2-00208-g9eaead5 #4
Call Trace:
 [<c025f212>] ? xfs_cmn_err+0x32/0x60
 [<c025f28e>] xfs_error_report+0x4e/0x50
 [<c024c132>] ? xfs_btree_check_block+0x32/0x40
 [<c024bfbd>] xfs_btree_check_lblock+0x4d/0x190
 [<c024c132>] ? xfs_btree_check_block+0x32/0x40
<...>
Assertion failed: cur->bc_btnum != XFS_BTNUM_BMAP ||
cur->bc_private.b.allocated == 0, file: fs/xfs/xfs_btree.c, line: 348
------------[ cut here ]------------
kernel BUG at fs/xfs/support/debug.c:81!
invalid opcode: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
last sysfs file: /sys/devices/platform/w83627hf.656/name
Modules linked in: w83627hf hwmon_vid i2c_nforce2
Pid: 2481, comm: pdflush Not tainted (2.6.28-rc2-00208-g9eaead5 #4)
EIP: 0060:[<c029d6ee>] EFLAGS: 00010282 CPU: 0
EIP is at assfail+0x1e/0x30
<..>
Call Trace:
 [<c024bcd0>] ? xfs_btree_del_cursor+0x60/0x80
 [<c02445be>] ? xfs_bmapi+0x48e/0x1c90
 [<c014b7a5>] ? __lock_acquire+0x2c5/0x1000
 [<c014b7a5>] ? __lock_acquire+0x2c5/0x1000
 [<c0270cd4>] ? xfs_iomap_write_allocate+0x254/0x450

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Dave Chinner]

On Mon, Jan 12, 2009 at 03:48:13AM +0300, Alexander Beregalov wrote:
> > Hmmmm - this might be getting closer to the source of the bug.
> > It's being detecting when reading in the buffer to do a left shift
> > now, not during the delete of a record.
> >
> > I'd suggest that you treat this as the same failure and continue
> > the bisect to try to find when no problems show up at all.
> 
> 687b890a184fef263ebb773926e1f4aa69240d01 is the first bad commit.

    [XFS] implement generic xfs_btree_lshift

    Make the btree left shift code generic. Based on a patch from David
    Chinner with lots of changes to follow the original btree implementations
    more closely. While this loses some of the generic helper routines for
    inserting/moving/removing records it also solves some of the one off bugs
    in the original code and makes it easier to verify.

> Does it make sense?

Yes, a bug in that patch could corrupt the btree in memory which we then trip
over later in delrec before it has been written to disk.

Thank you for isolating the problem to that commit - it greatly narrows down
the amount of code we need to search to find the bug. I'll have a look tonight
to see if I can spot the problem.

Cheers,

Dave.
-- 
Dave Chinner
david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

2009/1/12 Dave Chinner <david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org>:
> On Mon, Jan 12, 2009 at 03:48:13AM +0300, Alexander Beregalov wrote:
>> > Hmmmm - this might be getting closer to the source of the bug.
>> > It's being detecting when reading in the buffer to do a left shift
>> > now, not during the delete of a record.
>> >
>> > I'd suggest that you treat this as the same failure and continue
>> > the bisect to try to find when no problems show up at all.
>>
>> 687b890a184fef263ebb773926e1f4aa69240d01 is the first bad commit.
>
>    [XFS] implement generic xfs_btree_lshift
>
>    Make the btree left shift code generic. Based on a patch from David
>    Chinner with lots of changes to follow the original btree implementations
>    more closely. While this loses some of the generic helper routines for
>    inserting/moving/removing records it also solves some of the one off bugs
>    in the original code and makes it easier to verify.
>
>> Does it make sense?
>
> Yes, a bug in that patch could corrupt the btree in memory which we then trip
> over later in delrec before it has been written to disk.
>
> Thank you for isolating the problem to that commit - it greatly narrows down
> the amount of code we need to search to find the bug. I'll have a look tonight
> to see if I can spot the problem.

It seems 9eaead5 (implement generic xfs_btree_rshift) is really guilty, unless
the bug "XFS internal error xfs_btree_check_lblock at line 200 of file
fs/xfs/xfs_btree.c:"
which I posted 5 hours ago is completely different from the original
bug message.
I can not reproduce the bug on 278d0ca14.

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Dave Chinner]

On Mon, Jan 12, 2009 at 11:08:55AM +0300, Alexander Beregalov wrote:
> 2009/1/12 Dave Chinner <david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org>:
> > On Mon, Jan 12, 2009 at 03:48:13AM +0300, Alexander Beregalov wrote:
> >> > Hmmmm - this might be getting closer to the source of the bug.
> >> > It's being detecting when reading in the buffer to do a left shift
> >> > now, not during the delete of a record.
> >> >
> >> > I'd suggest that you treat this as the same failure and continue
> >> > the bisect to try to find when no problems show up at all.
> >>
> >> 687b890a184fef263ebb773926e1f4aa69240d01 is the first bad commit.
> >
> >    [XFS] implement generic xfs_btree_lshift
> >
> >    Make the btree left shift code generic. Based on a patch from David
> >    Chinner with lots of changes to follow the original btree implementations
> >    more closely. While this loses some of the generic helper routines for
> >    inserting/moving/removing records it also solves some of the one off bugs
> >    in the original code and makes it easier to verify.
> >
> >> Does it make sense?
> >
> > Yes, a bug in that patch could corrupt the btree in memory which we then trip
> > over later in delrec before it has been written to disk.
> >
> > Thank you for isolating the problem to that commit - it greatly narrows down
> > the amount of code we need to search to find the bug. I'll have a look tonight
> > to see if I can spot the problem.
> 
> It seems 9eaead5 (implement generic xfs_btree_rshift) is really guilty, unless
> the bug "XFS internal error xfs_btree_check_lblock at line 200 of file
> fs/xfs/xfs_btree.c:"
> which I posted 5 hours ago is completely different from the original
> bug message.
> I can not reproduce the bug on 278d0ca14.

Ok, Thanks for clarifying. ;)

I'll have a look through the rshift patch and try to find what
we broke by inspection.

Cheers,

Dave.
-- 
Dave Chinner
david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

Hi

Is it a new bug?
It is pre-2.6.29-rc1 kernel, which was supossed to be free of the bug
"fs_is_ok, file: fs/xfs/xfs_btree.c, line: 3327"

 Assertion failed: fsbno != NULLFSBLOCK, file: fs/xfs/xfs_btree.c, line: 816
 ------------[ cut here ]------------
 kernel BUG at fs/xfs/support/debug.c:81!
 invalid opcode: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
 last sysfs file: /sys/devices/platform/w83627hf.656/name
 Modules linked in: w83627hf hwmon_vid i2c_nforce2

 Pid: 10156, comm: pdflush Not tainted (2.6.28-rc2-00207-g278d0ca #5)
 EIP: 0060:[<c029dbee>] EFLAGS: 00010282 CPU: 0
 EIP is at assfail+0x1e/0x30
 EAX: 0000005f EBX: f7325a10 ECX: d3a24000 EDX: 00000000
 ESI: ffffffff EDI: 00000001 EBP: d3a25880 ESP: d3a25870
  DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
 Process pdflush (pid: 10156, ti=d3a24000 task=f6223cf0 task.ti=d3a24000)
 Stack:
  c04d6c04 c04b6a80 c04b6da8 00000330 d3a2589c c024c01e f6aca408 00cc8bc4
  00000001 ffffffff 00000001 d3a258b4 c024c0e5 f0a45000 f0a45000 00000000
  00000001 d3a258cc c024c161 00000003 d3a25a04 f0a45000 f0a45000 d3a2590c
 Call Trace:
  [<c024c01e>] ? xfs_btree_reada_bufl+0x2e/0x80
  [<c024c0e5>] ? xfs_btree_readahead_lblock+0x75/0x80
  [<c024c161>] ? xfs_btree_readahead+0x71/0x90
  [<c024ca2c>] ? xfs_btree_decrement+0x3c/0x2e0
  [<c024998c>] ? xfs_bmbt_delete+0x2c/0xa0
  [<c0240d24>] ? xfs_bmap_add_extent_delay_real+0x1674/0x1700
  [<c040e987>] ? _spin_unlock+0x27/0x50
  [<c022a0cb>] ? xfs_alloc_search_busy+0x8b/0xd0
  [<c0180cfc>] ? slab_pad_check+0x3c/0x120
  [<c022dc19>] ? xfs_alloc_vextent+0x2e9/0x760
  [<c01806bd>] ? check_object+0x13d/0x200
  [<c0241a86>] ? xfs_bmap_add_extent+0x626/0x670
  [<c0246bac>] ? xfs_bmbt_init_cursor+0x2c/0x100
  [<c02454cb>] ? xfs_bmapi+0xfcb/0x1c90
  [<c01806bd>] ? check_object+0x13d/0x200
  [<c02711d4>] ? xfs_iomap_write_allocate+0x254/0x450
  [<c02723d7>] ? xfs_iomap+0x3a7/0x3f0
  [<c02b32e6>] ? submit_bio+0x66/0x100
  [<c029117e>] ? xfs_map_blocks+0x3e/0x90
  [<c0292cda>] ? xfs_page_state_convert+0x2ea/0x740
  [<c029325e>] ? xfs_vm_writepage+0x5e/0xf0
  [<c016420b>] ? __writepage+0xb/0x40
  [<c01651b1>] ? write_cache_pages+0x1f1/0x330
  [<c0164200>] ? __writepage+0x0/0x40
  [<c0165313>] ? generic_writepages+0x23/0x30
  [<c0291133>] ? xfs_vm_writepages+0x43/0x50
  [<c02910f0>] ? xfs_vm_writepages+0x0/0x50
  [<c016534e>] ? do_writepages+0x2e/0x50
  [<c019ff72>] ? __writeback_single_inode+0x82/0x320
  [<c040e3c3>] ? _spin_lock+0x63/0x70
  [<c01a057a>] ? generic_sync_sb_inodes+0x23a/0x320
  [<c01a0836>] ? writeback_inodes+0x56/0xe0
  [<c01644cb>] ? wb_kupdate+0x7b/0xf0
  [<c01658f0>] ? pdflush+0x0/0x180
  [<c01659b0>] ? pdflush+0xc0/0x180
  [<c0164450>] ? wb_kupdate+0x0/0xf0
  [<c013b6ea>] ? kthread+0x3a/0x70
  [<c013b6b0>] ? kthread+0x0/0x70
  [<c0104157>] ? kernel_thread_helper+0x7/0x10
 Code: 00 e8 97 73 02 00 c9 c3 90 8d 74 26 00 55 89 e5 83 ec 10 89 4c
24 0c 89 54 24 08 89 44 24 04 c7 04 24 04 6c 4d c0 e8 fb de 16 00 <0f>
0b eb fe 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 89 e5
 EIP: [<c029dbee>] assfail+0x1e/0x30 SS:ESP 0068:d3a25870

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Eric Sandeen]

Alexander Beregalov wrote:
> Hi
> 
> Is it a new bug?
> It is pre-2.6.29-rc1 kernel, which was supossed to be free of the bug
> "fs_is_ok, file: fs/xfs/xfs_btree.c, line: 3327"

Do you have CONFIG_LBD on, and if not, can you enable it and retest?
hch narrowed a btree bug down to this...

-Eric

>  Assertion failed: fsbno != NULLFSBLOCK, file: fs/xfs/xfs_btree.c, line: 816
>  ------------[ cut here ]------------
>  kernel BUG at fs/xfs/support/debug.c:81!
>  invalid opcode: 0000 [#1] PREEMPT DEBUG_PAGEALLOC
>  last sysfs file: /sys/devices/platform/w83627hf.656/name
>  Modules linked in: w83627hf hwmon_vid i2c_nforce2
> 
>  Pid: 10156, comm: pdflush Not tainted (2.6.28-rc2-00207-g278d0ca #5)
>  EIP: 0060:[<c029dbee>] EFLAGS: 00010282 CPU: 0
>  EIP is at assfail+0x1e/0x30
>  EAX: 0000005f EBX: f7325a10 ECX: d3a24000 EDX: 00000000
>  ESI: ffffffff EDI: 00000001 EBP: d3a25880 ESP: d3a25870
>   DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
>  Process pdflush (pid: 10156, ti=d3a24000 task=f6223cf0 task.ti=d3a24000)
>  Stack:
>   c04d6c04 c04b6a80 c04b6da8 00000330 d3a2589c c024c01e f6aca408 00cc8bc4
>   00000001 ffffffff 00000001 d3a258b4 c024c0e5 f0a45000 f0a45000 00000000
>   00000001 d3a258cc c024c161 00000003 d3a25a04 f0a45000 f0a45000 d3a2590c
>  Call Trace:
>   [<c024c01e>] ? xfs_btree_reada_bufl+0x2e/0x80
>   [<c024c0e5>] ? xfs_btree_readahead_lblock+0x75/0x80
>   [<c024c161>] ? xfs_btree_readahead+0x71/0x90
>   [<c024ca2c>] ? xfs_btree_decrement+0x3c/0x2e0
>   [<c024998c>] ? xfs_bmbt_delete+0x2c/0xa0
>   [<c0240d24>] ? xfs_bmap_add_extent_delay_real+0x1674/0x1700
>   [<c040e987>] ? _spin_unlock+0x27/0x50
>   [<c022a0cb>] ? xfs_alloc_search_busy+0x8b/0xd0
>   [<c0180cfc>] ? slab_pad_check+0x3c/0x120
>   [<c022dc19>] ? xfs_alloc_vextent+0x2e9/0x760
>   [<c01806bd>] ? check_object+0x13d/0x200
>   [<c0241a86>] ? xfs_bmap_add_extent+0x626/0x670
>   [<c0246bac>] ? xfs_bmbt_init_cursor+0x2c/0x100
>   [<c02454cb>] ? xfs_bmapi+0xfcb/0x1c90
>   [<c01806bd>] ? check_object+0x13d/0x200
>   [<c02711d4>] ? xfs_iomap_write_allocate+0x254/0x450
>   [<c02723d7>] ? xfs_iomap+0x3a7/0x3f0
>   [<c02b32e6>] ? submit_bio+0x66/0x100
>   [<c029117e>] ? xfs_map_blocks+0x3e/0x90
>   [<c0292cda>] ? xfs_page_state_convert+0x2ea/0x740
>   [<c029325e>] ? xfs_vm_writepage+0x5e/0xf0
>   [<c016420b>] ? __writepage+0xb/0x40
>   [<c01651b1>] ? write_cache_pages+0x1f1/0x330
>   [<c0164200>] ? __writepage+0x0/0x40
>   [<c0165313>] ? generic_writepages+0x23/0x30
>   [<c0291133>] ? xfs_vm_writepages+0x43/0x50
>   [<c02910f0>] ? xfs_vm_writepages+0x0/0x50
>   [<c016534e>] ? do_writepages+0x2e/0x50
>   [<c019ff72>] ? __writeback_single_inode+0x82/0x320
>   [<c040e3c3>] ? _spin_lock+0x63/0x70
>   [<c01a057a>] ? generic_sync_sb_inodes+0x23a/0x320
>   [<c01a0836>] ? writeback_inodes+0x56/0xe0
>   [<c01644cb>] ? wb_kupdate+0x7b/0xf0
>   [<c01658f0>] ? pdflush+0x0/0x180
>   [<c01659b0>] ? pdflush+0xc0/0x180
>   [<c0164450>] ? wb_kupdate+0x0/0xf0
>   [<c013b6ea>] ? kthread+0x3a/0x70
>   [<c013b6b0>] ? kthread+0x0/0x70
>   [<c0104157>] ? kernel_thread_helper+0x7/0x10
>  Code: 00 e8 97 73 02 00 c9 c3 90 8d 74 26 00 55 89 e5 83 ec 10 89 4c
> 24 0c 89 54 24 08 89 44 24 04 c7 04 24 04 6c 4d c0 e8 fb de 16 00 <0f>
> 0b eb fe 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 89 e5
>  EIP: [<c029dbee>] assfail+0x1e/0x30 SS:ESP 0068:d3a25870
> 
> _______________________________________________
> xfs-masters mailing list
> xfs-masters-VZNHf3L845pBDgjK7y7TUQ@public.gmane.org
> http://oss.sgi.com/mailman/listinfo/xfs-masters
> 

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Christoph Hellwig]

On Tue, Jan 20, 2009 at 09:54:44PM +0300, Alexander Beregalov wrote:
> Hi
> 
> Is it a new bug?
> It is pre-2.6.29-rc1 kernel, which was supossed to be free of the bug
> "fs_is_ok, file: fs/xfs/xfs_btree.c, line: 3327"

From the trace it's a post-btree consolidation kernel, so all the bugs
in -rc1 are there, too.  It looks like this is the readahead type
cockup that Geert noticed, so this particular one should be fixed in
-rc2.

I would strongly recommend to just turn on CNFIG_LBD for post-2.6.28
kernels for now as all the problems showing up are without CONFIG_LBD.

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Alexander Beregalov]

2009/1/20 Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>:
> On Tue, Jan 20, 2009 at 09:54:44PM +0300, Alexander Beregalov wrote:
>> Hi
>>
>> Is it a new bug?
>> It is pre-2.6.29-rc1 kernel, which was supossed to be free of the bug
>> "fs_is_ok, file: fs/xfs/xfs_btree.c, line: 3327"
>
> From the trace it's a post-btree consolidation kernel, so all the bugs
> in -rc1 are there, too.  It looks like this is the readahead type
> cockup that Geert noticed, so this particular one should be fixed in
> -rc2.
>
> I would strongly recommend to just turn on CNFIG_LBD for post-2.6.28
> kernels for now as all the problems showing up are without CONFIG_LBD.
Yes, both bugs do not appear on kernel with CONFIG_LBD=y.

But I do not need LBD. Does XFS strongly require LBD?
Should I always turn it on even if I do not have files or devices of size 2Tb+ ?

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Christoph Hellwig]

On Fri, Jan 23, 2009 at 09:34:36PM +0300, Alexander Beregalov wrote:
> But I do not need LBD. Does XFS strongly require LBD?
> Should I always turn it on even if I do not have files or devices of size 2Tb+ ?

The bug is now fixed in the development tree, but until you upgrade
to a kernel with the fix enabling CONFIG_LBD will keep you from hitting
the bug.

As you unfortunately noticed the !CONFIG_LBD case doesn't really get
much test coverage, so I would personally recommend turning it on even
if you don't need it.

Re: [xfs-masters] 2.6.29-rc: kernel BUG at fs/xfs/support/debug.c:108

[Eric Sandeen]

Alexander Beregalov wrote:
> 2009/1/10 Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>:

>> How big is the filesystem where you see this corruption? Maybe I could
>> reproduce it locally with a xfs_metadump image.
> 
> They are big enough, I have four such FS's for >200Gb

An xfs_metadump should not be so large, though.

But bisecting as hch suggested (keepin the "working" header for each
bisection point) would be great.

-Eric