Re: 2.6.20-rc4-mm1

Re: 2.6.20-rc4-mm1

[Thomas Gleixner]

On Thu, 2007-01-11 at 22:26 -0800, Andrew Morton wrote:
> - Merged the "filesystem AIO patches".

Hotfixes alreday applied.

BUG: at /home/tglx/work/kernel/vanilla/linux-2.6.20-rc4-mm1/arch/i386/mm/highmem.c:60 kmap_atomic()
 [<c0105fba>] show_trace_log_lvl+0x1a/0x2f
 [<c01065ff>] show_trace+0x12/0x14
 [<c01066b1>] dump_stack+0x16/0x18
 [<c011fad8>] kmap_atomic+0x12f/0x1c8
 [<f88db29c>] ata_scsi_rbuf_get+0x22/0x37 [libata]
 [<f88db773>] atapi_qc_complete+0x1ee/0x240 [libata]
 [<f88d666b>] __ata_qc_complete+0x86/0x8d [libata]
 [<f88d670a>] ata_qc_complete+0x98/0x9e [libata]
 [<f88d98f4>] ata_qc_complete_multiple+0x8a/0xa4 [libata]
 [<f88b3a2c>] ahci_interrupt+0x2bd/0x3b9 [ahci]
 [<c0154a2d>] handle_IRQ_event+0x21/0x48
 [<c0155971>] handle_edge_irq+0xd1/0x115
 [<c01071af>] do_IRQ+0x6c/0x89
 [<c0105a0b>] common_interrupt+0x23/0x28
 [<c01031c2>] mwait_idle+0xd/0xf
 [<c010343f>] cpu_idle+0xb7/0xf1
 [<c010141d>] rest_init+0x37/0x3a
 [<c04629bd>] start_kernel+0x3cc/0x3ef
 [<00000000>] 0x0

ata_scsi_rbuf_get requests KM_IRQ0 type memory and calls kmap_atomic
with interrupts enabled.

Boot proceeds, but gets stuck hard at:
"Remounting root filesystem in read-write mode:"

No SysRq-T, nothing.

The above BUG seems unrelated to that. Investigating further.

	tglx

Re: 2.6.20-rc4-mm1

[Thomas Gleixner]

On Sun, 2007-01-14 at 10:48 +0100, Thomas Gleixner wrote:
> ata_scsi_rbuf_get requests KM_IRQ0 type memory and calls kmap_atomic
> with interrupts enabled.
> 
> Boot proceeds, but gets stuck hard at:
> "Remounting root filesystem in read-write mode:"
> 
> No SysRq-T, nothing.
> 
> The above BUG seems unrelated to that. Investigating further.

Bisect identified: git-block.patch

	tglx

Re: 2.6.20-rc4-mm1

[Thomas Gleixner]

On Sun, 2007-01-14 at 11:46 +0100, Thomas Gleixner wrote:
> > Boot proceeds, but gets stuck hard at:
> > "Remounting root filesystem in read-write mode:"
> > 
> > No SysRq-T, nothing.
> > 
> > The above BUG seems unrelated to that. Investigating further.
> 
> Bisect identified: git-block.patch

Does only happen on 2 systems. Both have sata + raid1 setup. I managed 
to get a stacktrace from the SMP box. Sits there and sleeps forever.

	tglx

[<c032ac64>] io_schedule+0x7a/0x9a
[<c0157f89>] sleep_on_page+0x8/0xc
[<c032ae45>] __wait_on_bit+0x36/0x5d
[<c01580d8>] wait_on_page_bit+0x5b/0x61
[<c0158a2b>] wait_on_page_writeback_range+0x4f/0xef
[<c0158b0f>] filemap_fdatawait+0x44/0x49
[<c0158da0>] filemap_write_and_wait+0x22/0x2d
[<c0190e39>] sync_blockdev+0x17/0x1d
[<c01a27af>] quota_sync_sb+0x33/0xd6
[<c01a2874>] sync_dquots+0x22/0xfa
[<c01757cf>] __fsync_super+0x17/0x66
[<c0175829>] fsync_super+0xb/0x19
[<c0175880>] do_remount_sb+0x49/0x101
[<c0187f98>] do_mount+0x1ad/0x678
[<c01884d2>] sys_mount+0x6f/0xa4
[<c0103f6a>] sysenter_past_esp+0x5f/0x99

Re: 2.6.20-rc4-mm1

[Jens Axboe]

On Sun, Jan 14 2007, Thomas Gleixner wrote:
> On Sun, 2007-01-14 at 11:46 +0100, Thomas Gleixner wrote:
> > > Boot proceeds, but gets stuck hard at:
> > > "Remounting root filesystem in read-write mode:"
> > > 
> > > No SysRq-T, nothing.
> > > 
> > > The above BUG seems unrelated to that. Investigating further.
> > 
> > Bisect identified: git-block.patch
> 
> Does only happen on 2 systems. Both have sata + raid1 setup. I managed 
> to get a stacktrace from the SMP box. Sits there and sleeps forever.
> 
> 	tglx
> 
> [<c032ac64>] io_schedule+0x7a/0x9a
> [<c0157f89>] sleep_on_page+0x8/0xc
> [<c032ae45>] __wait_on_bit+0x36/0x5d
> [<c01580d8>] wait_on_page_bit+0x5b/0x61
> [<c0158a2b>] wait_on_page_writeback_range+0x4f/0xef
> [<c0158b0f>] filemap_fdatawait+0x44/0x49
> [<c0158da0>] filemap_write_and_wait+0x22/0x2d
> [<c0190e39>] sync_blockdev+0x17/0x1d
> [<c01a27af>] quota_sync_sb+0x33/0xd6
> [<c01a2874>] sync_dquots+0x22/0xfa
> [<c01757cf>] __fsync_super+0x17/0x66
> [<c0175829>] fsync_super+0xb/0x19
> [<c0175880>] do_remount_sb+0x49/0x101
> [<c0187f98>] do_mount+0x1ad/0x678
> [<c01884d2>] sys_mount+0x6f/0xa4
> [<c0103f6a>] sysenter_past_esp+0x5f/0x99

raid seems to have severe problems with the plugging change. I'll try
and find Neil and have a chat with him, hopefully we can work it out.

-- 
Jens Axboe

Re: 2.6.20-rc4-mm1

[Thomas Gleixner]

On Mon, 2007-01-15 at 09:05 +1100, Jens Axboe wrote:
> raid seems to have severe problems with the plugging change. I'll try
> and find Neil and have a chat with him, hopefully we can work it out.

Some hints:

mount(1899): WRITE block 16424 on md3
call md_write_start
md3_raid1(438): WRITE block 40965504 on sdb6
md3_raid1(438): WRITE block 40965504 on sda6
First Write sector 16424 disks 2

Stuck.

Note, that neither end_buffer_async_write() nor
raid1_end_write_request() are invoked, 

In a previous write invoked by:
fsck.ext3(1896): WRITE block 8552 on sdb1
end_buffer_async_write() is invoked.

sdb1 is not a part of a raid device.

Hope that helps,

	tglx

Re: 2.6.20-rc4-mm1

[Jens Axboe]

On Sun, Jan 14 2007, Thomas Gleixner wrote:
> On Mon, 2007-01-15 at 09:05 +1100, Jens Axboe wrote:
> > raid seems to have severe problems with the plugging change. I'll try
> > and find Neil and have a chat with him, hopefully we can work it out.
> 
> Some hints:
> 
> mount(1899): WRITE block 16424 on md3
> call md_write_start
> md3_raid1(438): WRITE block 40965504 on sdb6
> md3_raid1(438): WRITE block 40965504 on sda6
> First Write sector 16424 disks 2
> 
> Stuck.
> 
> Note, that neither end_buffer_async_write() nor
> raid1_end_write_request() are invoked, 
> 
> In a previous write invoked by:
> fsck.ext3(1896): WRITE block 8552 on sdb1
> end_buffer_async_write() is invoked.
> 
> sdb1 is not a part of a raid device.

When I briefly tested this before I left (and found it broken), doing a
cat /proc/mdstat got things going again. Hard if that's your rootfs,
it's just a hint :-)

> Hope that helps,

I can reproduce, so that's not a problem. I can't do much about it until
I'm back next week, but Neil might be able to help. We shall see. Thanks
for testing.

-- 
Jens Axboe

Re: 2.6.20-rc4-mm1

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> > In a previous write invoked by: fsck.ext3(1896): WRITE block 8552 on 
> > sdb1 end_buffer_async_write() is invoked.
> > 
> > sdb1 is not a part of a raid device.
> 
> When I briefly tested this before I left (and found it broken), doing 
> a cat /proc/mdstat got things going again. Hard if that's your rootfs, 
> it's just a hint :-)

hm, so you knew it's broken, still you let Andrew pick it up, or am i 
misunderstanding something?

	Ingo

Re: 2.6.20-rc4-mm1

[Jens Axboe]

On Mon, Jan 15 2007, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > > In a previous write invoked by: fsck.ext3(1896): WRITE block 8552 on 
> > > sdb1 end_buffer_async_write() is invoked.
> > > 
> > > sdb1 is not a part of a raid device.
> > 
> > When I briefly tested this before I left (and found it broken), doing 
> > a cat /proc/mdstat got things going again. Hard if that's your rootfs, 
> > it's just a hint :-)
> 
> hm, so you knew it's broken, still you let Andrew pick it up, or am i 
> misunderstanding something?

Well the raid issue wasn't known before it was in -mm.

-- 
Jens Axboe

Re: 2.6.20-rc4-mm1

[Jens Axboe]

On Sun, Jan 14 2007, Thomas Gleixner wrote:
> On Sun, 2007-01-14 at 11:46 +0100, Thomas Gleixner wrote:
> > > Boot proceeds, but gets stuck hard at:
> > > "Remounting root filesystem in read-write mode:"
> > > 
> > > No SysRq-T, nothing.
> > > 
> > > The above BUG seems unrelated to that. Investigating further.
> > 
> > Bisect identified: git-block.patch
> 
> Does only happen on 2 systems. Both have sata + raid1 setup. I managed 
> to get a stacktrace from the SMP box. Sits there and sleeps forever.
> 
> 	tglx
> 
> [<c032ac64>] io_schedule+0x7a/0x9a
> [<c0157f89>] sleep_on_page+0x8/0xc
> [<c032ae45>] __wait_on_bit+0x36/0x5d
> [<c01580d8>] wait_on_page_bit+0x5b/0x61
> [<c0158a2b>] wait_on_page_writeback_range+0x4f/0xef
> [<c0158b0f>] filemap_fdatawait+0x44/0x49
> [<c0158da0>] filemap_write_and_wait+0x22/0x2d
> [<c0190e39>] sync_blockdev+0x17/0x1d
> [<c01a27af>] quota_sync_sb+0x33/0xd6
> [<c01a2874>] sync_dquots+0x22/0xfa
> [<c01757cf>] __fsync_super+0x17/0x66
> [<c0175829>] fsync_super+0xb/0x19
> [<c0175880>] do_remount_sb+0x49/0x101
> [<c0187f98>] do_mount+0x1ad/0x678
> [<c01884d2>] sys_mount+0x6f/0xa4
> [<c0103f6a>] sysenter_past_esp+0x5f/0x99

raid seems to have severe problems with the plugging change. I'll try
and find Neil and have a chat with him, hopefully we can work it out.

-- 
Jens Axboe