From mboxrd@z Thu Jan 1 00:00:00 1970 From: "liuqing@huayun.com" Subject: Re: Multipath ID not equal to LUN scsi ID Date: Wed, 12 Jul 2017 10:14:55 +0800 Message-ID: <2017071210145556280876@huayun.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4340623080079712622==" Return-path: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: maier , dm-devel Cc: liuqing List-Id: dm-devel.ids This is a multi-part message in MIME format. --===============4340623080079712622== Content-Type: multipart/alternative; boundary="----=_001_NextPart321453017436_=----" This is a multi-part message in MIME format. ------=_001_NextPart321453017436_=---- Content-Type: text/plain; charset="GB2312" Content-Transfer-Encoding: base64 VGhhbmtzIFN0ZWZmZW4uDQoNCk91ciBtYW5hZ2VtZW50IHRvb2woT3BlbnN0YWNrKSB3aWxsIGRv IGFzIHRoZSBjbGVhbiBhcyB0aGUgZG9jdW1lbnQoaHR0cHM6Ly9hY2Nlc3MucmVkaGF0LmNvbS9k b2N1bWVudGF0aW9uL2VuLVVTL1JlZF9IYXRfRW50ZXJwcmlzZV9MaW51eC83L2h0bWwvU3RvcmFn ZV9BZG1pbmlzdHJhdGlvbl9HdWlkZS9yZW1vdmluZ19kZXZpY2VzLmh0bWwpDQppbiBub3JtYWwg c2l0dWF0aW9uLCBidXQgT3BlbnN0YWNrIGRvIGhhdmUgYnVncyB3aGljaCB3aWxsIGxlYWQgdG8g cGF0aHMgb2YgdW5tYXBwZWQgTFVOIG5vdCBiZWVuIHJlbW92ZWQuIFRoaXMgd2lsbCBjYXVzZSBk YXRhIGNydXNoIGlmIHRoZSBvcmlnaW5hbCBwYXRocyBhcmUgYWRkZWQgYmFjay4gDQpBbHRob3Vn aCBPcGVuc3RhY2sgc2hvdWxkIG5vdCBsZWF2ZSB0aGUgcGF0aHMsIG11bHRpcGF0aCBzaG91bGQg Zm91bmQgb3V0IGRpZmZlcmVudCBMVU5zIGJ5IGRpZmZlcmVudCBJRF9TRVJJQUwuIEFueSBGQyBh ZG1pbmlzdG9yIG1heSBmb3JnZXQgdG8gZGVsZXRlIHRoZSBwYXRocyBiZWZvcmUgYWRkaW5nIGEg bmV3IExVTiB3aXRoIG9sZCBMVU4gSUQuDQoNCkZsdXNoIHRoZSBwYXRoIGFuZCByZXNjYW4gc2Nz aSBidXMgYW5kIHRoZW4gZG8gYSBtdWx0aXBhdGggLXYyIGNvdWxkIG1ha2UgdGhlIG5ldyBwYXRo IGN1cnJlY3QuIEkgYW0gdGhpbmtpbmcgd2hldGhlciB3ZSBjb3VsZCBhZGQgc2ltaWxhciBzdGVw cyBpbiBtdWx0aXBhdGgtdG9vbCB3aGVuIG11bHRpcGF0aCBkZXRlY3RlZCBhIG9mZmxpbmVkIHBh dGggYmVjb21lDQpvbmxpbmUobm8gdWRldiBldmVudCBhYm91dCBuZXcgYXJyaXZlZCBMVU4gaW4g dGhpcyBzaXR1YXRpb24pLiAgTm90IGFuIGV4cGVydCBvZiB0aGUgbXVsdGlwYXRoIGNvZGUsIGRv IHlvdSB0aGluayB0aGlzIGlzIHJlYXNvbmFibGU/DQoNCg0KPT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT0NClNpbmNlIExpbnV4IGRvZXMgbm90IHJlYWxseSBh dXRvbWF0aWNhbGx5IGhhbmRsZSB0YXJnZXQgdm9sdW1lDQpyZW1hcHBpbmcsIHlvdSBtaWdodCBi ZSBiZXR0ZXIgb2Ygd2l0aCBleHBsaWNpdGx5IHRlYXJpbmcgZG93biB0aGUgb2xkDQp2b2x1bWUg aW4gTGludXggYmVmb3JlIGFueSB1bm1hcC9yZW1hcCBvbiB0aGUgdGFyZ2V0LiBUaGlzIGlzIGFs cmVhZHkNCnRyaWNreSBlbm91Z2ggdG8gZ2V0IHJpZ2h0Lg0KIA0KaHR0cHM6Ly9hY2Nlc3MucmVk aGF0LmNvbS9kb2N1bWVudGF0aW9uL2VuLVVTL1JlZF9IYXRfRW50ZXJwcmlzZV9MaW51eC83L2h0 bWwvU3RvcmFnZV9BZG1pbmlzdHJhdGlvbl9HdWlkZS9yZW1vdmluZ19kZXZpY2VzLmh0bWwNCiAN CiANClJlc2l6aW5nIG9mIHRoZSBzYW1lIHZvbHVtZSBvbiB0aGUgdGFyZ2V0IGlzIGEgc3BlY2lh bCBjYXNlIHdoaWNoIHNob3VsZA0Kd29yaywgYnV0IGV2ZW4gdGhhdCBvbmUgaW5jbHVkZXMgYSBy ZXNjYW4gb2YgdGhlIGFmZmVjdGVkIHNjc2kgZGV2aWNlcw0KKHBhdGhzKSBhcyBwcmUtcmVxLg0K IA0KaHR0cHM6Ly9hY2Nlc3MucmVkaGF0LmNvbS9kb2N1bWVudGF0aW9uL2VuLVVTL1JlZF9IYXRf RW50ZXJwcmlzZV9MaW51eC83L2h0bWwvU3RvcmFnZV9BZG1pbmlzdHJhdGlvbl9HdWlkZS9vbmxp bmUtaXNjc2ktcmVzaXppbmcuaHRtbCNyZXNpemluZy1mYy1sb2dpY2FsLXVuaXRzDQpodHRwczov L2FjY2Vzcy5yZWRoYXQuY29tL2RvY3VtZW50YXRpb24vZW4tVVMvUmVkX0hhdF9FbnRlcnByaXNl X0xpbnV4LzcvaHRtbC9TdG9yYWdlX0FkbWluaXN0cmF0aW9uX0d1aWRlL29ubGluZS1pc2NzaS1y ZXNpemluZy5odG1sI2lkbTgzNzMyNjQNCmh0dHBzOi8vYWNjZXNzLnJlZGhhdC5jb20vZG9jdW1l bnRhdGlvbi9lbi1VUy9SZWRfSGF0X0VudGVycHJpc2VfTGludXgvNy9odG1sL0RNX011bHRpcGF0 aC9vbmxpbmVfZGV2aWNlX3Jlc2l6ZS5odG1sDQogDQogDQpOQjogTm9uZSBvZiB0aGUgVVJMcyBw cm92aWRlZCBhcmUgc3BlY2lmaWMgdG8gYSBkaXN0cm8gb3IgZGV2aWNlIGRyaXZlci4NClRob3Nl IHdlcmUganVzdCBleGFtcGxlIGRvY3VtZW50YXRpb24gcmVmZXJlbmNlcyBJIGNvdWxkIGZpbmQg cXVpY2tseS4NCiANCk9uIDA3LzExLzIwMTcgMTE6MDcgQU0sIFN0ZWZmZW4gTWFpZXIgd3JvdGU6 DQo+IE9uIDA3LzExLzIwMTcgMDU6NDkgQU0sIGxpdXFpbmdAaHVheXVuLmNvbSB3cm90ZToNCj4+ IE9uY2UgYSBuZXcgYXJyaXZlZCBMVU4gbWFwcGVkIHdlIHdpbGwgZG8gcmVzY2FuIGJ5ICJlY2hv ICctIC0gLScgPg0KPj4gL3N5cy9jbGFzcy9zY3NpX2hvc3QvaG9zdDIvc2NhbiIuDQo+DQo+IEkg d2FzIHJlZmVycmluZyB0byBhIHJlc2NhbiBvZiB0aGUgc2NzaSBkZXZpY2UsDQo+DQo+IGh0dHBz Oi8vd3d3LmlibS5jb20vc3VwcG9ydC9rbm93bGVkZ2VjZW50ZXIvbGludXhvbmlibS9jb20uaWJt LmxpbnV4LnoubGhkZC9saGRkX3RfZmNwX3dya19yZXNjYW4uaHRtbA0KPg0KPg0KPiBub3QgYSBz Y2FuIG9mIHRoZSBTY3NpX0hvc3QuDQo+IFRoZSBsYXR0ZXIgbGlrZWx5IGtlZXBzIHRoZSBvbGQg c2NzaSBkZXZpY2Ugd2l0aCBpdHMgb2xkIHByb3BlcnRpZXMNCj4gYmVpbmcgb2JsaXZpb3VzIHRv IHRoZSB2b2x1bWUgcmVtYXBwaW5nIG9uIHRoZSBzdG9yYWdlIHRhcmdldDsgaXQncw0KPiBhYm91 dCBkaXNjb3ZlcmluZyBuZXcgc2NzaSBkZXZpY2VzIHByZXZpb3VzbHkgbm90IGJlaW5nIGtub3du IHRvIExpbnV4Lg0KPg0KPj4gQWZ0ZXIgcmVzY2FuIG9ubHkgdGhlIHNjc2lfaWQgdG9vbCBnaXZl IHRoZSByaWdodCBzZXJpYWwgaWQsIHVkZXZhZG0NCj4+IHN0aWxsIGdvdCB0aGUgcHJ2aW91cyBv bmUuDQo+PiBJIGhhdmUgbW9uaXRvciB0aGUgdWRldiBldmVudCBieSB1ZGV2YWRtIG1vbml0b3Ig d2hpbGUgbWFwcGluZyBhIG5ldw0KPj4gTFVOIHRvIHRoZSBob3N0LCB3aG8gd2lsbCByZXVzZSB0 aGUgb3JpZ2luYWwgcGF0aC4gTm8gYWRkIGV2ZW50IGlzDQo+PiB0cmlnZ2VycmVkLCBvbmx5IGRt LVggZW1pdHMgYSBjaGFuZ2UgZXZlbnQuDQo+PiBCdXQgaWYgdGhlIG9yaWdpbmFsIHBhdGggaXMg ZGVsZXRlZChyZW1vdmVkKSB0aGVuIGFkZCBldmVudCB3aWxsIGJlDQo+PiB0cmlnZ2VycmVkLg0K Pj4NCj4+IEZsdXNoIHRoZSBvbGQgV1dJRCBjb3VsZCBtYWtlIHRoZSBXV0lEIGNvcnJlY3QgYnV0 IHRoZSBzaXplIGlzIHN0aWxsDQo+PiBpbmNvcnJlY3QgYXMgZm9sbG93aW5nLiBBbmQgMzYwMDUw NzYzMDA4MTBlYWRmODAwMDAwMDAwMDAwMTU1IGlzDQo+PiBhY3R1YWxseSA1R0IuDQo+DQo+IGxp a2VseSBzYW1lIHJlYXNvbiBhcyBhYm92ZQ0KPg0KPj4gSSBidWlsdCB0aGUgdG9vbCB1c2luZyBs YXRlc3QgY29kZSBhbmQgdHJpZWQgYm90aCBhdHRyaWJ1dGVfaWQgYW5kDQo+PiBnZXR1aWRfY2Fs bG91dC4gVGhlIGlzc3VlIGV4aXN0IGluIGJvdGggY29uZmlndXJhdGlvbi4NCj4+DQo+Pg0KPj4+ IERlYXIgbGlzdCwNCj4+PiBXZSBoYXZlIGEgRkMgc3RvcmFnZSBhbmQgdXNpbmcgbXVsdGlwYXRo ZCB0byBtYW5hZ2VyIHRoZSBGQyBwYXRocy4NCj4+PiBJJ3ZlIG1ldCBhbiBpc3N1ZSBpbiB0aGlz IGVudmlyb25tZW50LiBUaGUgZm9sbG93aW5nIGlzIGhvdyB0bw0KPj4+IHJlY3JlYXRlIHRoZSBp c3N1ZS4NCj4+Pg0KPj4+ID09PT09PT0NCj4+PiAxLiBNYXAgYSBMVU4gdG8gaG9zdCB3aXRoIExV TiBJRCAwLA0KPj4+IDIuIHJlc2NhbiBmY19ob3N0LCBhIG5ldyBwYXRoIHdpbGwgYmUgZm91bmQg YnkgbXVsdGlwYXRoLg0KPj4+IDMuIFVubWFwIExVTiAwLiAgcGF0aCB3aWxsIGZhaWxlZCBhcyBm b2xsb3dpbmcuDQo+Pj4gW3Jvb3RAbG9jYWxob3N0IHN5c10jIG11bHRpcGF0aCAtbGwNCj4+PiBK dWwgMTAgMTg6NDE6NTAgfCBzZHA6IGNvdWxkbid0IGdldCBhc3ltbWV0cmljIGFjY2VzcyBzdGF0 ZQ0KPj4+IEp1bCAxMCAxODo0MTo1MCB8IHNkcTogY291bGRuJ3QgZ2V0IGFzeW1tZXRyaWMgYWNj ZXNzIHN0YXRlDQo+Pj4gMzYwMDUwNzYzMDA4MTBlYWRmODAwMDAwMDAwMDAwMTU2IGRtLTMgSUJN LDIxNDUNCj4+PiBzaXplPTguMEcgZmVhdHVyZXM9JzIgcXVldWVfaWZfbm9fcGF0aCByZXRhaW5f YXR0YWNoZWRfaHdfaGFuZGxlcicNCj4+PiBod2hhbmRsZXI9JzEgYWx1YScgd3A9cncNCj4+PiB8 LSstIHBvbGljeT0nc2VydmljZS10aW1lIDAnIHByaW89MCBzdGF0dXM9ZW5hYmxlZA0KPj4+IHwg YC0gMjowOjA6MCBzZHAgODoyNDAgZmFpbGVkIGZhdWx0eSBydW5uaW5nDQo+Pj4gYC0rLSBwb2xp Y3k9J3NlcnZpY2UtdGltZSAwJyBwcmlvPTAgc3RhdHVzPWVuYWJsZWQNCj4+PiAgICAgYC0gMjow OjE6MCBzZHEgNjU6MCAgZmFpbGVkIGZhdWx0eSBydW5uaW5nDQo+Pj4gNC4gTWFwIGFub3RoZXIg TFVOIHdoaWNoIGhhdmUgZGlmZmVyZW50IElEX1NFUklBTCBidXQgd2l0aCB0aGUgc2FtZQ0KPj4+ IExVTiBJRCgwKS4NCj4+IERpZCB5b3UgInJlc2NhbiIgdGhlIFNDU0kgZGV2aWNlIHZpYSBzeXNm cyB0byBsZXQgTGludXgga25vdyB0aGF0IHRoaXMNCj4+IGlzIG5vdyBpbiBmYWN0IGEgZGlmZmVy ZW50IGRldmljZT8NCj4+IEFGQUlLLCBMaW51eCBkZWNvZGVzIFNDU0kgc2Vuc2UgZGF0YSBmb3Ig TFVOcyByZW1hcHBlZCBvbiB0aGUgc3RvcmFnZQ0KPj4gdGFyZ2V0IGFuZCBlbWl0cyBhIHVkZXYg ZXZlbnQsIGJ1dCBJJ20gbm90IGF3YXJlIG9mIGFueSBkZWZhdWx0IHVkZXYNCj4+IHJ1bGUgdGhh dCB3b3VsZCBhY3R1YWxseSByZWFjdC4gVGhlIGtlcm5lbCBpdHNlbGYgZG9lcyBub3QgcmVhY3Qg b3RoZXINCj4+IHRoYW4gY3JlYXRpbmcgdGhlIHVldmVudC4NCj4NCj4NCiANCi0tDQpNaXQgZnJl dW5kbGljaGVuIEdyPz9lbiAvIEtpbmQgcmVnYXJkcw0KU3RlZmZlbiBNYWllcg0KIA0KTGludXgg b24geiBTeXN0ZW1zIERldmVsb3BtZW50DQogDQpJQk0gRGV1dHNjaGxhbmQgUmVzZWFyY2ggJiBE ZXZlbG9wbWVudCBHbWJIDQpWb3JzaXR6ZW5kZSBkZXMgQXVmc2ljaHRzcmF0czogTWFydGluYSBL b2VkZXJpdHoNCkdlc2NoYWVmdHNmdWVocnVuZzogRGlyayBXaXR0a29wcA0KU2l0eiBkZXIgR2Vz ZWxsc2NoYWZ0OiBCb2VibGluZ2VuDQpSZWdpc3RlcmdlcmljaHQ6IEFtdHNnZXJpY2h0IFN0dXR0 Z2FydCwgSFJCIDI0MzI5NA0KDQoNCmxpdXFpbmdAaHVheXVuLmNvbQ0K ------=_001_NextPart321453017436_=---- Content-Type: text/html; charset="GB2312" Content-Transfer-Encoding: quoted-printable =0A
Than= ks Steffen.

in normal situation, but Openstack do have bugs which wil= l lead to paths of unmapped LUN not been removed. This will cause data crush if the= original paths are added back. 
Although Openstack should not leave the = paths, multipath should found out different LUNs by different ID_SERIAL. A= ny FC administor may forget to delete the paths before adding a new LUN wi= th old LUN ID.

Flush the path and rescan scsi bus and then do a multipath -v2= could make the new path currect. I am thinking whether we could add simil= ar steps in multipath-tool when multipath detected a offlined path become<= /span>
onli= ne(no udev event about new arrived LUN in this situation).  Not an expert of the multipath code, do = you think this is reasonable?
=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=0A
S= ince Linux does not really automatically handle target volume
remapping, you might be better of with explicitly teari= ng down the old
volume in Linux before any = unmap/remap on the target. This is already
= tricky enough to get right.
 
<= div style=3D"font-family: =CE=A2=C8=ED=D1=C5=BA=DA, Tahoma; font-size: 14p= x; line-height: normal;">https://acce= ss.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_= Administration_Guide/removing_devices.html
 
 
Resizing of the same volume on the target is a special case which sh= ould
work, but even that one includes a res= can of the affected scsi devices
(paths) as= pre-req.
 
<= div style=3D"font-family: =CE=A2=C8=ED=D1=C5=BA=DA, Tahoma; font-size: 14p= x; line-height: normal;"> 
 
NB: None of the URLs provided are specific to = a distro or device driver.
Those were just = example documentation references I could find quickly.
 
On 07/11/2017 11:07 AM= , Steffen Maier wrote:
> On 07/11/2017 = 05:49 AM, liuqing@huayun.com wrote:
>>= ; Once a new arrived LUN mapped we will do rescan by "echo '- - -' >
>> /sys/class/scsi_host/host2/scan".
>
> I = was referring to a rescan of the scsi device,
>
>
>
> not a scan of the= Scsi_Host.
> The latter likely keeps th= e old scsi device with its old properties
&= gt; being oblivious to the volume remapping on the storage target; it's
> about discovering new scsi devices previ= ously not being known to Linux.
>
=
>> After rescan only the scsi_id tool give= the right serial id, udevadm
>> stil= l got the prvious one.
>> I have mon= itor the udev event by udevadm monitor while mapping a new
>> LUN to the host, who will reuse the original path= . No add event is
>> triggerred, only= dm-X emits a change event.
>> But = if the original path is deleted(removed) then add event will be
>> triggerred.
&= gt;>
>> Flush the old WWID could m= ake the WWID correct but the size is still
= >> incorrect as following. And 36005076300810eadf800000000000155 is<= /div>
>> actually 5GB.
>
> likely same reason as = above
>
&= gt;> I built the tool using latest code and tried both attribute_id and=
>> getuid_callout. The issue exist i= n both configuration.
>>
>>
>>> = Dear list,
>>> We have a FC storag= e and using multipathd to manager the FC paths.
>>> I've met an issue in this environment. The following is = how to
>>> recreate the issue.
>>>
&= gt;>> =3D=3D=3D=3D=3D=3D=3D
>>= ;> 1. Map a LUN to host with LUN ID 0,
&= gt;>> 2. rescan fc_host, a new path will be found by multipath.
>>> 3. Unmap LUN 0.  path will fa= iled as following.
>>> [root@loc= alhost sys]# multipath -ll
>>> Jul= 10 18:41:50 | sdp: couldn't get asymmetric access state
>>> Jul 10 18:41:50 | sdq: couldn't get asymmetri= c access state
>>> 36005076300810e= adf800000000000156 dm-3 IBM,2145
>>&g= t; size=3D8.0G features=3D'2 queue_if_no_path retain_attached_hw_handler'<= /div>
>>> hwhandler=3D'1 alua' wp=3Drw
>>> |-+- policy=3D'service-time 0' = prio=3D0 status=3Denabled
>>> | `= - 2:0:0:0 sdp 8:240 failed faulty running
&= gt;>> `-+- policy=3D'service-time 0' prio=3D0 status=3Denabled
=
>>>     `- 2:0:1:0 = sdq 65:0  failed faulty running
>= >> 4. Map another LUN which have different ID_SERIAL but with the sa= me
>>> LUN ID(0).
>> Did you "rescan" the SCSI device via sysfs to let= Linux know that this
>> is now in = fact a different device?
>> AFAIK, = Linux decodes SCSI sense data for LUNs remapped on the storage
>> target and emits a udev event, but I'm not aw= are of any default udev
>> rule that = would actually react. The kernel itself does not react other
>> than creating the uevent.
>
>
 
--
Mit freundlichen Gr??en / Kind regards
 
Linux on z Systems Development
&= nbsp;
IBM Deutschland Research & Develo= pment GmbH
Vorsitzende des Aufsichtsrats: M= artina Koederitz
Geschaeftsfuehrung: Dirk W= ittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB = 243294
=0A
liuqing@huayun.com
=0A ------=_001_NextPart321453017436_=------ --===============4340623080079712622== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============4340623080079712622==--