From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756096AbbDJOcQ (ORCPT ); Fri, 10 Apr 2015 10:32:16 -0400 Received: from mx2.parallels.com ([199.115.105.18]:45282 "EHLO mx2.parallels.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755224AbbDJOcM (ORCPT ); Fri, 10 Apr 2015 10:32:12 -0400 From: James Bottomley To: "calvinowens@fb.com" CC: "linux-kernel@vger.kernel.org" , "MPT-FusionLinux.pdl@avagotech.com" , "kernel-team@fb.com" , "stable@vger.kernel.org" , "praveen.krishnamoorthy@avagotech.com" , "abhijit.mahajan@avagotech.com" , "nagalakshmi.nandigama@avagotech.com" , "sreekanth.reddy@avagotech.com" Subject: Re: [PATCH] mpt2sas: mpt3sas: Fix memory corruption during initialization Thread-Topic: [PATCH] mpt2sas: mpt3sas: Fix memory corruption during initialization Thread-Index: AQHQc14VdT+o2jX7SEej7dBx6WSGbp1GxNyA Date: Fri, 10 Apr 2015 14:30:54 +0000 Message-ID: <1428676329.2178.2.camel@Odin.com> References: <1428650094-12750-1-git-send-email-calvinowens@fb.com> In-Reply-To: <1428650094-12750-1-git-send-email-calvinowens@fb.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [50.46.149.214] Content-Type: text/plain; charset="utf-8" Content-ID: <7268849EA7624741ACEE757A93CCC1A0@sw.swsoft.com> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id t3AEWMRw024818 On Fri, 2015-04-10 at 00:14 -0700, Calvin Owens wrote: > While _scsih_probe_sas() is initializing devices, the hardware can trigger a > MPI2_EVENT_SAS_TOPO_RC_TARG_NOT_RESPONDING interrupt. > > The handler for TARG_NOT_RESPONDING calls _scsih_device_remove_by_handle(), > which deletes the device in question from either ioc->sas_device_list or > ioc->sas_device_init_list. Since _scsih_probe_sas() uses no exclusion when > iterating over ioc->sas_device_init_list, this results in a use-after-free > in _scsih_probe_sas(), and also corrupts the list: > > mpt2sas1: removing handle(0x0020), sas_addr(0x5f80f418573360e0) > mpt2sas1: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000) > ------------[ cut here ]------------ > WARNING: at lib/list_debug.c:56 __list_del_entry+0xc3/0xd0() > list_del corruption, ffff88240012fa00->prev is LIST_POISON2 (dead000000200200) > > Workqueue: events work_for_cpu_fn > ffffffff810c4f17 ffff881214825b38 0000000000000009 ffff881214825ae8 > ffffffff8169b61e ffff881214825b28 ffffffff8104a990 0000000000000002 > ffff88240012f900 ffff88240012fa00 ffff881217595af8 0000000000000282 > Call Trace: > [] ? print_modules+0xd7/0x120 > [] dump_stack+0x19/0x1b > [] warn_slowpath_common+0x70/0xa0 > [] warn_slowpath_fmt+0x46/0x50 > [] ? _raw_spin_lock_irqsave+0x84/0xa0 > [] ? _scsih_probe_sas+0x8e/0x110 [mpt2sas] > [] __list_del_entry+0xc3/0xd0 > [] _scsih_probe_sas+0x99/0x110 [mpt2sas] > [] _scsih_scan_finished+0x19f/0x2c0 [mpt2sas] > [] do_scsi_scan_host+0x77/0xa0 > [] scsi_scan_host+0x190/0x1c0 > [] _scsih_probe+0x452/0x640 [mpt2sas] > [] local_pci_probe+0x4b/0x80 > [] work_for_cpu_fn+0x18/0x30 > [] process_one_work+0x212/0x6e0 > [] ? process_one_work+0x1a6/0x6e0 > [] ? local_clock+0x4f/0x60 > [] process_scheduled_works+0x2c/0x40 > [] worker_thread+0x262/0x370 > [] ? rescuer_thread+0x360/0x360 > [] kthread+0xdb/0xe0 > [] ? trace_hardirqs_on+0xd/0x10 > [] ? kthread_create_on_node+0x140/0x140 > [] ret_from_fork+0x7c/0xb0 > [] ? kthread_create_on_node+0x140/0x140 > ---[ end trace 41352a0bd2d0d61b ]--- > > This either results in an immediate panic, or corrupts random memory and > causes nasty problems later in the box's uptime. > > This patch splices the discovered devices out of the global list while > holding the lock, since _scsih_probe_sas() always removes them from that > global list anyway (either deleting them if initialization fails, or > moving them onto ioc->sas_device_list if it succeeds). The interrupt that > caused this bug will no longer cause the device to be removed during > initialization, since it won't exist on the global lists, but > _scsih_probe_sas() will remove it anyway when it fails to initialize. Hopefully the avago team will curate this, but just in case they don't, the correct list to make sure it gets the attention of storage people should be linux-scsi@vger.kernel.org James {.n++%ݶw{.n+{G{ayʇڙ,jfhz_(階ݢj"mG?&~iOzv^m ?I From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: James Bottomley To: "calvinowens@fb.com" CC: "linux-kernel@vger.kernel.org" , "MPT-FusionLinux.pdl@avagotech.com" , "kernel-team@fb.com" , "stable@vger.kernel.org" , "praveen.krishnamoorthy@avagotech.com" , "abhijit.mahajan@avagotech.com" , "nagalakshmi.nandigama@avagotech.com" , "sreekanth.reddy@avagotech.com" Subject: Re: [PATCH] mpt2sas: mpt3sas: Fix memory corruption during initialization Date: Fri, 10 Apr 2015 14:30:54 +0000 Message-ID: <1428676329.2178.2.camel@Odin.com> References: <1428650094-12750-1-git-send-email-calvinowens@fb.com> In-Reply-To: <1428650094-12750-1-git-send-email-calvinowens@fb.com> Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-ID: <7268849EA7624741ACEE757A93CCC1A0@sw.swsoft.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: T24gRnJpLCAyMDE1LTA0LTEwIGF0IDAwOjE0IC0wNzAwLCBDYWx2aW4gT3dlbnMgd3JvdGU6DQo+ IFdoaWxlIF9zY3NpaF9wcm9iZV9zYXMoKSBpcyBpbml0aWFsaXppbmcgZGV2aWNlcywgdGhlIGhh cmR3YXJlIGNhbiB0cmlnZ2VyIGENCj4gTVBJMl9FVkVOVF9TQVNfVE9QT19SQ19UQVJHX05PVF9S RVNQT05ESU5HIGludGVycnVwdC4NCj4gDQo+IFRoZSBoYW5kbGVyIGZvciBUQVJHX05PVF9SRVNQ T05ESU5HIGNhbGxzIF9zY3NpaF9kZXZpY2VfcmVtb3ZlX2J5X2hhbmRsZSgpLA0KPiB3aGljaCBk ZWxldGVzIHRoZSBkZXZpY2UgaW4gcXVlc3Rpb24gZnJvbSBlaXRoZXIgaW9jLT5zYXNfZGV2aWNl X2xpc3Qgb3INCj4gaW9jLT5zYXNfZGV2aWNlX2luaXRfbGlzdC4gU2luY2UgX3Njc2loX3Byb2Jl X3NhcygpIHVzZXMgbm8gZXhjbHVzaW9uIHdoZW4NCj4gaXRlcmF0aW5nIG92ZXIgaW9jLT5zYXNf ZGV2aWNlX2luaXRfbGlzdCwgdGhpcyByZXN1bHRzIGluIGEgdXNlLWFmdGVyLWZyZWUNCj4gaW4g X3Njc2loX3Byb2JlX3NhcygpLCBhbmQgYWxzbyBjb3JydXB0cyB0aGUgbGlzdDoNCj4gDQo+ICAg bXB0MnNhczE6IHJlbW92aW5nIGhhbmRsZSgweDAwMjApLCBzYXNfYWRkcigweDVmODBmNDE4NTcz MzYwZTApDQo+ICAgbXB0MnNhczE6IGxvZ19pbmZvKDB4MzExMTEwMDApOiBvcmlnaW5hdG9yKFBM KSwgY29kZSgweDExKSwgc3ViX2NvZGUoMHgxMDAwKQ0KPiAgIC0tLS0tLS0tLS0tLVsgY3V0IGhl cmUgXS0tLS0tLS0tLS0tLQ0KPiAgIFdBUk5JTkc6IGF0IGxpYi9saXN0X2RlYnVnLmM6NTYgX19s aXN0X2RlbF9lbnRyeSsweGMzLzB4ZDAoKQ0KPiAgIGxpc3RfZGVsIGNvcnJ1cHRpb24sIGZmZmY4 ODI0MDAxMmZhMDAtPnByZXYgaXMgTElTVF9QT0lTT04yIChkZWFkMDAwMDAwMjAwMjAwKQ0KPiAg IDxzbmlwPg0KPiAgIFdvcmtxdWV1ZTogZXZlbnRzIHdvcmtfZm9yX2NwdV9mbg0KPiAgICBmZmZm ZmZmZjgxMGM0ZjE3IGZmZmY4ODEyMTQ4MjViMzggMDAwMDAwMDAwMDAwMDAwOSBmZmZmODgxMjE0 ODI1YWU4DQo+ICAgIGZmZmZmZmZmODE2OWI2MWUgZmZmZjg4MTIxNDgyNWIyOCBmZmZmZmZmZjgx MDRhOTkwIDAwMDAwMDAwMDAwMDAwMDINCj4gICAgZmZmZjg4MjQwMDEyZjkwMCBmZmZmODgyNDAw MTJmYTAwIGZmZmY4ODEyMTc1OTVhZjggMDAwMDAwMDAwMDAwMDI4Mg0KPiAgIENhbGwgVHJhY2U6 DQo+ICAgIFs8ZmZmZmZmZmY4MTBjNGYxNz5dID8gcHJpbnRfbW9kdWxlcysweGQ3LzB4MTIwDQo+ ICAgIFs8ZmZmZmZmZmY4MTY5YjYxZT5dIGR1bXBfc3RhY2srMHgxOS8weDFiDQo+ICAgIFs8ZmZm ZmZmZmY4MTA0YTk5MD5dIHdhcm5fc2xvd3BhdGhfY29tbW9uKzB4NzAvMHhhMA0KPiAgICBbPGZm ZmZmZmZmODEwNGFhNzY+XSB3YXJuX3Nsb3dwYXRoX2ZtdCsweDQ2LzB4NTANCj4gICAgWzxmZmZm ZmZmZjgxNmEyMjM0Pl0gPyBfcmF3X3NwaW5fbG9ja19pcnFzYXZlKzB4ODQvMHhhMA0KPiAgICBb PGZmZmZmZmZmYTAwMTBlOGU+XSA/IF9zY3NpaF9wcm9iZV9zYXMrMHg4ZS8weDExMCBbbXB0MnNh c10NCj4gICAgWzxmZmZmZmZmZjgxMzJhNWEzPl0gX19saXN0X2RlbF9lbnRyeSsweGMzLzB4ZDAN Cj4gICAgWzxmZmZmZmZmZmEwMDEwZTk5Pl0gX3Njc2loX3Byb2JlX3NhcysweDk5LzB4MTEwIFtt cHQyc2FzXQ0KPiAgICBbPGZmZmZmZmZmYTAwMTFkNWY+XSBfc2NzaWhfc2Nhbl9maW5pc2hlZCsw eDE5Zi8weDJjMCBbbXB0MnNhc10NCj4gICAgWzxmZmZmZmZmZjgxNDI5ZDY3Pl0gZG9fc2NzaV9z Y2FuX2hvc3QrMHg3Ny8weGEwDQo+ICAgIFs8ZmZmZmZmZmY4MTQyOWYyMD5dIHNjc2lfc2Nhbl9o b3N0KzB4MTkwLzB4MWMwDQo+ICAgIFs8ZmZmZmZmZmZhMDAxMTQwMj5dIF9zY3NpaF9wcm9iZSsw eDQ1Mi8weDY0MCBbbXB0MnNhc10NCj4gICAgWzxmZmZmZmZmZjgxMzQ0NGViPl0gbG9jYWxfcGNp X3Byb2JlKzB4NGIvMHg4MA0KPiAgICBbPGZmZmZmZmZmODEwNmI4NDg+XSB3b3JrX2Zvcl9jcHVf Zm4rMHgxOC8weDMwDQo+ICAgIFs8ZmZmZmZmZmY4MTA3MDAxMj5dIHByb2Nlc3Nfb25lX3dvcmsr MHgyMTIvMHg2ZTANCj4gICAgWzxmZmZmZmZmZjgxMDZmZmE2Pl0gPyBwcm9jZXNzX29uZV93b3Jr KzB4MWE2LzB4NmUwDQo+ICAgIFs8ZmZmZmZmZmY4MTA4ZWQxZj5dID8gbG9jYWxfY2xvY2srMHg0 Zi8weDYwDQo+ICAgIFs8ZmZmZmZmZmY4MTA3MDUwYz5dIHByb2Nlc3Nfc2NoZWR1bGVkX3dvcmtz KzB4MmMvMHg0MA0KPiAgICBbPGZmZmZmZmZmODEwNzBhZTI+XSB3b3JrZXJfdGhyZWFkKzB4MjYy LzB4MzcwDQo+ICAgIFs8ZmZmZmZmZmY4MTA3MDg4MD5dID8gcmVzY3Vlcl90aHJlYWQrMHgzNjAv MHgzNjANCj4gICAgWzxmZmZmZmZmZjgxMDc4ZjNiPl0ga3RocmVhZCsweGRiLzB4ZTANCj4gICAg WzxmZmZmZmZmZjgxMGI1ZThkPl0gPyB0cmFjZV9oYXJkaXJxc19vbisweGQvMHgxMA0KPiAgICBb PGZmZmZmZmZmODEwNzhlNjA+XSA/IGt0aHJlYWRfY3JlYXRlX29uX25vZGUrMHgxNDAvMHgxNDAN Cj4gICAgWzxmZmZmZmZmZjgxNmFjMDFjPl0gcmV0X2Zyb21fZm9yaysweDdjLzB4YjANCj4gICAg WzxmZmZmZmZmZjgxMDc4ZTYwPl0gPyBrdGhyZWFkX2NyZWF0ZV9vbl9ub2RlKzB4MTQwLzB4MTQw DQo+ICAgLS0tWyBlbmQgdHJhY2UgNDEzNTJhMGJkMmQwZDYxYiBdLS0tDQo+IA0KPiBUaGlzIGVp dGhlciByZXN1bHRzIGluIGFuIGltbWVkaWF0ZSBwYW5pYywgb3IgY29ycnVwdHMgcmFuZG9tIG1l bW9yeSBhbmQNCj4gY2F1c2VzIG5hc3R5IHByb2JsZW1zIGxhdGVyIGluIHRoZSBib3gncyB1cHRp bWUuDQo+IA0KPiBUaGlzIHBhdGNoIHNwbGljZXMgdGhlIGRpc2NvdmVyZWQgZGV2aWNlcyBvdXQg b2YgdGhlIGdsb2JhbCBsaXN0IHdoaWxlDQo+IGhvbGRpbmcgdGhlIGxvY2ssIHNpbmNlIF9zY3Np aF9wcm9iZV9zYXMoKSBhbHdheXMgcmVtb3ZlcyB0aGVtIGZyb20gdGhhdA0KPiBnbG9iYWwgbGlz dCBhbnl3YXkgKGVpdGhlciBkZWxldGluZyB0aGVtIGlmIGluaXRpYWxpemF0aW9uIGZhaWxzLCBv cg0KPiBtb3ZpbmcgdGhlbSBvbnRvIGlvYy0+c2FzX2RldmljZV9saXN0IGlmIGl0IHN1Y2NlZWRz KS4gVGhlIGludGVycnVwdCB0aGF0DQo+IGNhdXNlZCB0aGlzIGJ1ZyB3aWxsIG5vIGxvbmdlciBj YXVzZSB0aGUgZGV2aWNlIHRvIGJlIHJlbW92ZWQgZHVyaW5nDQo+IGluaXRpYWxpemF0aW9uLCBz aW5jZSBpdCB3b24ndCBleGlzdCBvbiB0aGUgZ2xvYmFsIGxpc3RzLCBidXQNCj4gX3Njc2loX3By b2JlX3NhcygpIHdpbGwgcmVtb3ZlIGl0IGFueXdheSB3aGVuIGl0IGZhaWxzIHRvIGluaXRpYWxp emUuDQoNCkhvcGVmdWxseSB0aGUgYXZhZ28gdGVhbSB3aWxsIGN1cmF0ZSB0aGlzLCBidXQganVz dCBpbiBjYXNlIHRoZXkgZG9uJ3QsDQp0aGUgY29ycmVjdCBsaXN0IHRvIG1ha2Ugc3VyZSBpdCBn ZXRzIHRoZSBhdHRlbnRpb24gb2Ygc3RvcmFnZSBwZW9wbGUNCnNob3VsZCBiZQ0KDQpsaW51eC1z Y3NpQHZnZXIua2VybmVsLm9yZw0KDQpKYW1lcw0KDQo=