From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751263Ab1HJEbK (ORCPT ); Wed, 10 Aug 2011 00:31:10 -0400 Received: from TYO201.gate.nec.co.jp ([202.32.8.193]:57379 "EHLO tyo201.gate.nec.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750850Ab1HJEbJ (ORCPT ); Wed, 10 Aug 2011 00:31:09 -0400 Message-ID: <4E420941.3080601@ce.jp.nec.com> Date: Wed, 10 Aug 2011 13:29:53 +0900 From: "Jun'ichi Nomura" User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110428 Fedora/3.1.10-1.fc14 Thunderbird/3.1.10 MIME-Version: 1.0 To: James.Bottomley@HansenPartnership.com, jaxboe@fusionio.com CC: roland@purestorage.com, stern@rowland.harvard.edu, linux-scsi@vger.kernel.org, "linux-kernel@vger.kernel.org" , device-mapper development , Kiyoshi Ueda Subject: [BUG] Oops when SCSI device under multipath is removed Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi James, With the attached shell script, blk_insert_cloned_request will cause oops with 3.1-rc1. (This is a regression introduced in 2.6.39) The regression was introduced by 86cbfb5607d4b81b1a993ff689bbd2addd5d3a9b, "[SCSI] put stricter guards on queue dead checks". Part of the commit has moved scsi_free_queue(), which calls blk_cleanup_queue(), to scsi_remove_device(), which is called while the device is still open. The oops occurs because blk_insert_cloned_request() is called after blk_cleanup_queue() is called (which frees elevator and turns on QUEUE_FLAG_DEAD). 2 patches have been proposed but neither of them included: 1) Add QUEUE_FLAG_DEAD check in blk_insert_cloned_request() https://lkml.org/lkml/2011/7/8/457 2) SCSI to call blk_cleanup_queue() from device's ->release() callback (before 2.6.39, it used to work like this) https://lkml.org/lkml/2011/7/2/106 Both work fine for this test case but it seems it's not possible to make the patch 1) safe because QUEUE_FLAG_DEAD check is racy. There is a window between the check and the use of elevator. So I think the patch 2) is better. Could you please consider to include it? BUG: unable to handle kernel NULL pointer dereference at (null) IP: [< (null)>] (null) PGD 29c015067 PUD 29855a067 PMD 0 Oops: 0010 [#1] SMP CPU 2 .. Pid: 6125, comm: dd Not tainted 3.1.0-rc1 #1 RIP: 0010:[<0000000000000000>] [< (null)>] (null) RSP: 0018:ffff880288377990 EFLAGS: 00010092 RAX: ffff88029c3e4e00 RBX: ffff88029a324cc8 RCX: 0000000000000240 RDX: 0000000000000002 RSI: 0000000000000001 RDI: ffff88029a324cc8 RBP: ffff8802883779a8 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000002 R11: 0000000000000000 R12: ffff88029a324cc8 R13: 0000000000000002 R14: 0000000000000002 R15: ffff88029a46c180 FS: 00007fc4daf3d700(0000) GS:ffff8802afa00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000029abe4000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process dd (pid: 6125, threadinfo ffff880288376000, task ffff8802879ac260) Stack: ffffffff81223d8a ffff8802883779d8 ffff88029b5a6018 ffff8802883779d8 ffffffff81224bdf ffff88029b5a6018 ffff88029a324cc8 0000000000000002 ffff88029a46c000 ffff880288377a08 ffffffff8122974d ffff880288377a08 Call Trace: [] ? elv_drain_elevator+0x2a/0x80 [] __elv_add_request+0x9f/0x200 [] add_acct_request+0x3d/0x50 [] blk_insert_cloned_request+0x65/0x90 [] dm_dispatch_request+0x3e/0x70 [dm_mod] [] dm_request_fn+0x191/0x290 [dm_mod] [] __blk_run_queue+0x1e/0x20 [] queue_unplugged+0x4e/0xc0 [] blk_flush_plug_list+0x1b6/0x210 [] io_schedule+0x75/0xd0 [] __blockdev_direct_IO+0x924/0xb20 [] blkdev_direct_IO+0x57/0x60 [] ? blkdev_get_block+0x70/0x70 [] generic_file_aio_read+0x6f5/0x770 [] ? _raw_spin_unlock+0x2b/0x40 [] ? handle_pte_fault+0x40a/0x9c0 [] ? native_sched_clock+0x13/0x60 [] ? sched_clock+0x9/0x10 [] do_sync_read+0xda/0x120 [] ? security_file_permission+0x8e/0x90 [] vfs_read+0xcd/0x190 [] sys_read+0x54/0xa0 [] system_call_fastpath+0x16/0x1b Code: Bad RIP value. RIP [< (null)>] (null) RSP CR2: 0000000000000000 ---[ end trace 4abd28efc271910a ]--- -- Jun'ichi Nomura, NEC Corporation #!/bin/bash dev=$1 if [ -z "$dev" ]; then echo "usage: $0 where is unused SCSI device" exit 1 fi if [ ! -e /dev/$dev -o ! -d /sys/block/$dev ] ; then echo "device $dev not found" exit 1 fi if [ $dev = ${dev#sd} ]; then echo "device $dev is not SCSI device" exit 1 fi echo "Use SCSI device $dev" mapname=deadmp maptab="0 $(blockdev --getsz /dev/$dev) multipath 0 0 1 1 round-robin 1 1 1 1 /dev/$dev 1" modprobe dm-multipath modprobe dm-round-robin echo "Creating multipath device: $mapname" echo $maptab | dmsetup create $mapname if [ $? -ne 0 ]; then echo "failed to create mpath device" exit 1 fi echo "Deleting $dev" echo 1 > /sys/block/$dev/device/delete sleep 1 echo "Try I/O on $dev" dd if=/dev/mapper/${mapname} of=/dev/null bs=4k count=1 iflag=direct sleep 10 dmsetup remove $mapname