Re: Soft lockups in mballoc code

[Mingming Cao <cmm@us.ibm.com>]

On Mon, 2008-05-12 at 13:21 -0400, Nick Dokos wrote:
> I keep getting soft lockups in mballoc code, running fsstress.
> I'm running a 2.6.25.2 kernel, with 1.40.8 e2fsprogs. Here
> is sample console output on a 2-cpu Opteron box with a 2Tb filesystem -
> the two stack traces keep repeating every minute or so and the only
> way to get out is to reset the box:
> 
Nick, thanks for reporting this. I remember saw soft lockup issue with
mballoc on 2.6.25-rc kernels.  Could you run the latest 2.6.26-rc2
kernel? There are lots of bug fixes have been pushed to 2.6.26-rc1.
Aneesh, do you know if this is a new issue?

Mingming
> ---------------------------------------------------------------------------
> [root@dl385 fsstress]# /home/nick/src/fsstress/fsstress -d /mnt2/fsstress -l 10 -n 100 -p 20
> seed = 1211090909
> BUG: soft lockup - CPU#1 stuck for 61s! [fsstress:3974]
> CPU 1:
> Modules linked in: qla2xxx ext4dev jbd2 crc16 ipv6 autofs4 hidp rfcomm l2cap bluetooth sunrpc cpufreq_ondemand dm_mirror dm_multipath dm_mod sbs sbshc battery acpi_memhotplug ac parport_pc lp parport sg floppy ide_cd_mod serio_raw cdrom button tg3 amd_rng k8temp shpchp i2c_amd756 pcspkr hwmon i2c_core scsi_transport_fc mptspi mptscsih scsi_transport_spi mptbase cciss sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd [last unloaded: qla2xxx]
> Pid: 3974, comm: fsstress Not tainted 2.6.25.2 #1
> RIP: 0010:[<ffffffff88412790>]  [<ffffffff88412790>] :ext4dev:ext4_mb_new_blocks+0x7bd/0x1618
> RSP: 0018:ffff8101f304b898  EFLAGS: 00000202
> RAX: 00000000ffffffff RBX: ffff8101f4c86918 RCX: 0000000000000007
> RDX: ffff8100f31d35c0 RSI: 00000000000029fd RDI: ffffe200034f6548
> RBP: 00000000000014fe R08: 0000000000000000 R09: ffff8101f4cbd280
> R10: 0000000000000001 R11: ffffffff884174c0 R12: 0000000000000f00
> R13: ffff8100f2de8b58 R14: ffffffff883feadd R15: ffff8101f4c622b8
> FS:  00007f78917726f0(0000) GS:ffff8101ff6b3f40(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000003dca80b9a0 CR3: 00000001f3036000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> 
> Call Trace:
>  [<ffffffff8841274d>] ? :ext4dev:ext4_mb_new_blocks+0x77a/0x1618
>  [<ffffffff8022779f>] ? set_next_entity+0x23/0x73
>  [<ffffffff8026000c>] ? compat_irq_chip_set_default_handler+0x5/0x11
>  [<ffffffff883fbd99>] ? :ext4dev:ext4_new_blocks+0x6a/0x88
>  [<ffffffff883fefb5>] ? :ext4dev:ext4_get_blocks_handle+0x355/0x84f
>  [<ffffffff883ff773>] ? :ext4dev:ext4_get_block+0x0/0xf9
>  [<ffffffff802aa74d>] ? __find_get_block+0x148/0x158
>  [<ffffffff883ff5c0>] ? :ext4dev:ext4_get_blocks_wrap+0x111/0x135
>  [<ffffffff883ff82d>] ? :ext4dev:ext4_get_block+0xba/0xf9
>  [<ffffffff802ab509>] ? __block_prepare_write+0x19d/0x36c
>  [<ffffffff883ff773>] ? :ext4dev:ext4_get_block+0x0/0xf9
>  [<ffffffff802ab775>] ? block_write_begin+0x78/0xc9
>  [<ffffffff88400d67>] ? :ext4dev:ext4_write_begin+0xeb/0x1b1
>  [<ffffffff883ff773>] ? :ext4dev:ext4_get_block+0x0/0xf9
>  [<ffffffff80292ab5>] ? __page_symlink+0x44/0xf7
>  [<ffffffff8840491d>] ? :ext4dev:ext4_symlink+0x136/0x1db
>  [<ffffffff80293915>] ? vfs_symlink+0xbf/0x12e
>  [<ffffffff80295d27>] ? sys_symlinkat+0x87/0xd4
>  [<ffffffff8025cbdf>] ? audit_syscall_exit+0x2e4/0x303
>  [<ffffffff8020bdae>] ? tracesys+0x71/0xe1
>  [<ffffffff8020be19>] ? tracesys+0xdc/0xe1
> 
> BUG: soft lockup - CPU#0 stuck for 61s! [fsstress:3984]
> CPU 0:
> Modules linked in: qla2xxx ext4dev jbd2 crc16 ipv6 autofs4 hidp rfcomm l2cap bluetooth sunrpc cpufreq_ondemand dm_mirror dm_multipath dm_mod sbs sbshc battery acpi_memhotplug ac parport_pc lp parport sg floppy ide_cd_mod serio_raw cdrom button tg3 amd_rng k8temp shpchp i2c_amd756 pcspkr hwmon i2c_core scsi_transport_fc mptspi mptscsih scsi_transport_spi mptbase cciss sd_mod scsi_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd [last unloaded: qla2xxx]
> Pid: 3984, comm: fsstress Not tainted 2.6.25.2 #1
> RIP: 0010:[<ffffffff884145e0>]  [<ffffffff884145e0>] :ext4dev:ext4_mb_discard_inode_preallocations+0x1c6/0x27b
> RSP: 0018:ffff8101f3521e18  EFLAGS: 00000202
> RAX: 00000000ffffffff RBX: ffff8100f2def470 RCX: 0000000000000007
> RDX: ffff8100f31d35c0 RSI: 00000000000014fe RDI: ffff8100f2dbd470
> RBP: 0000000000000000 R08: ffff81000900ec80 R09: 0000000000001000
> R10: ffff8100f2debae8 R11: ffff8101fc9d8000 R12: ffff8101f4c8bcc0
> R13: 0000000000001000 R14: 0000000000000000 R15: 000000000a7f0000
> FS:  00007f78917726f0(0000) GS:ffffffff805b9000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000003dca8c2200 CR3: 00000001f4023000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> 
> Call Trace:
>  [<ffffffff88414591>] :ext4dev:ext4_mb_discard_inode_preallocations+0x177/0x27b
>  [<ffffffff883fbe50>] :ext4dev:ext4_discard_reservation+0x24/0x68
>  [<ffffffff883fc872>] :ext4dev:ext4_release_file+0x2d/0x6f
>  [<ffffffff8028d243>] __fput+0xb1/0x17e
>  [<ffffffff8028ab77>] filp_close+0x5d/0x65
>  [<ffffffff8028bcee>] sys_close+0x7e/0xb7
>  [<ffffffff8020be19>] tracesys+0xdc/0xe1
> 
> 
> 
> ...
> 
> ---------------------------------------------------------------------------
> 
> Known bug? Is there a fix somewhere?
> 
> Let me know if more information is needed.
> 
> Thanks very much,
> Nick
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html