From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?B?QkVSVFJBTkQgSm/Dq2w=?= Subject: Re: [BUG] Raid1/5 over iSCSI trouble Date: Fri, 19 Oct 2007 23:04:15 +0200 Message-ID: <47191BCF.2000908@systella.fr> References: <4714BB92.7040701@systella.fr> <47161CE3.80909@systella.fr> <47181CB2.1060602@tmr.com> <471864F8.9010209@systella.fr> <1192809103.30976.11.camel@dwillia2-linux.ch.intel.com> <4718DE66.8000905@tmr.com> <471916B5.6080709@systella.fr> <47191855.4020402@systella.fr> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <47191855.4020402@systella.fr> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iscsitarget-devel-bounces@lists.sourceforge.net Errors-To: iscsitarget-devel-bounces@lists.sourceforge.net To: Bill Davidsen Cc: linux-raid@vger.kernel.org, sparclinux@vger.kernel.org, Dan Williams , iscsitarget-devel@lists.sourceforge.net List-Id: linux-raid.ids QkVSVFJBTkQgSm/Dq2wgd3JvdGU6Cj4gQkVSVFJBTkQgSm/Dq2wgd3JvdGU6Cj4+IEJpbGwgRGF2 aWRzZW4gd3JvdGU6Cj4+PiBEYW4gV2lsbGlhbXMgd3JvdGU6Cj4+Pj4gT24gRnJpLCAyMDA3LTEw LTE5IGF0IDAxOjA0IC0wNzAwLCBCRVJUUkFORCBKb8OrbCB3cm90ZToKPj4+PiAgCj4+Pj4+ICAg ICAgICAgSSBydW4gZm9yIDEyIGhvdXJzIHNvbWUgZGQncyAocmVhZCBhbmQgd3JpdGUgaW4gbnVs bGlvKQo+Pj4+PiBiZXR3ZWVuCj4+Pj4+IGluaXRpYXRvciBhbmQgdGFyZ2V0IHdpdGhvdXQgYW55 IGRpc2Nvbm5lY3Rpb24uIFRodXMgaVNDU0kgY29kZSBzZWVtcwo+Pj4+PiB0bwo+Pj4+PiBiZSBy b2J1c3QuIEJvdGggaW5pdGlhdG9yIGFuZCB0YXJnZXQgYXJlIGFsb25lIG9uIGEgc2luZ2xlIGdp Z2FiaXQKPj4+Pj4gZXRoZXJuZXQgbGluayAod2l0aG91dCBhbnkgc3dpdGNoKS4gSSdtIGludmVz dGlnYXRpbmcuLi4KPj4+Pj4gICAgIAo+Pj4+Cj4+Pj4gQ2FuIHlvdSByZXByb2R1Y2Ugb24gMi42 LjIyPwo+Pj4+Cj4+Pj4gQWxzbywgSSBkbyBub3QgdGhpbmsgdGhpcyBpcyB0aGUgY2F1c2Ugb2Yg eW91ciBmYWlsdXJlLCBidXQgeW91IGhhdmUKPj4+PiBDT05GSUdfRE1BX0VOR0lORT15IGluIHlv dXIgY29uZmlnLiAgU2V0dGluZyB0aGlzIHRvICduJyB3aWxsIGNvbXBpbGUKPj4+PiBvdXQgdGhl IHVubmVlZGVkIGNoZWNrcyBmb3Igb2ZmbG9hZCBlbmdpbmVzIGluIGFzeW5jX21lbWNweSBhbmQK Pj4+PiBhc3luY194b3IuCj4+Pgo+Pj4gR2l2ZW4gdGhhdCBvZmZsb2FkIGVuZ2luZXMgYXJlIGZh ciBsZXNzIHRlc3RlZCBjb2RlLCBJIHRoaW5rIHRoaXMgaXMgCj4+PiBhIHZlcnkgZ29vZCB0aGlu ZyB0byB0cnkhCj4+Cj4+ICAgICBJJ20gdHJ5aW5nIHdpaHRvdXQgQ09ORklHX0RNQV9FTkdJTkU9 eS4gaXN0ZDEgb25seSB1c2VzIDQwJSBvZiBvbmUgCj4+IENQVSB3aGVuIEkgcmVidWlsZCBteSBy YWlkMSBhcnJheS4gMSUgb2YgdGhpcyBhcnJheSB3YXMgbm93IAo+PiByZXN5bmNocm9uaXplZCB3 aXRob3V0IGFueSBoYW5nLgo+Pgo+PiBSb290IGdlcnNod2luOlsvdXNyL3NjcmlwdHNdID4gY2F0 IC9wcm9jL21kc3RhdAo+PiBQZXJzb25hbGl0aWVzIDogW3JhaWQxXSBbcmFpZDZdIFtyYWlkNV0g W3JhaWQ0XQo+PiBtZDcgOiBhY3RpdmUgcmFpZDEgc2RpMVsyXSBtZF9kMHAxWzBdCj4+ICAgICAg IDE0NjQ3MjU2MzIgYmxvY2tzIFsyLzFdIFtVX10KPj4gICAgICAgWz4uLi4uLi4uLi4uLi4uLi4u Li4uLl0gIHJlY292ZXJ5ID0gIDEuMCUgKDE1NzA1NTM2LzE0NjQ3MjU2MzIpIAo+PiBmaW5pc2g9 MTEwMy45bWluIHNwZWVkPTIxODc1Sy9zZWMKPiAKPiAgICAgU2FtZSByZXN1bHQuLi4KPiAKPiBj b25uZWN0aW9uMjowOiBpc2NzaTogZGV0ZWN0ZWQgY29ubiBlcnJvciAoMTAxMSkKPiAKPiAgICAg ICAgICBzZXNzaW9uMjogaXNjc2k6IHNlc3Npb24gcmVjb3ZlcnkgdGltZWQgb3V0IGFmdGVyIDEy MCBzZWNzCj4gc2QgNDowOjA6MDogc2NzaTogRGV2aWNlIG9mZmxpbmVkIC0gbm90IHJlYWR5IGFm dGVyIGVycm9yIHJlY292ZXJ5Cj4gc2QgNDowOjA6MDogc2NzaTogRGV2aWNlIG9mZmxpbmVkIC0g bm90IHJlYWR5IGFmdGVyIGVycm9yIHJlY292ZXJ5Cj4gc2QgNDowOjA6MDogc2NzaTogRGV2aWNl IG9mZmxpbmVkIC0gbm90IHJlYWR5IGFmdGVyIGVycm9yIHJlY292ZXJ5Cj4gc2QgNDowOjA6MDog c2NzaTogRGV2aWNlIG9mZmxpbmVkIC0gbm90IHJlYWR5IGFmdGVyIGVycm9yIHJlY292ZXJ5Cj4g c2QgNDowOjA6MDogc2NzaTogRGV2aWNlIG9mZmxpbmVkIC0gbm90IHJlYWR5IGFmdGVyIGVycm9y IHJlY292ZXJ5Cj4gc2QgNDowOjA6MDogc2NzaTogRGV2aWNlIG9mZmxpbmVkIC0gbm90IHJlYWR5 IGFmdGVyIGVycm9yIHJlY292ZXJ5Cj4gc2QgNDowOjA6MDogc2NzaTogRGV2aWNlIG9mZmxpbmVk IC0gbm90IHJlYWR5IGFmdGVyIGVycm9yIHJlY292ZXJ5CgoJU29ycnkgZm9yIHRoaXMgbGFzdCBt YWlsLiBJIGhhdmUgZm91bmQgYW5vdGhlciBtaXN0YWtlLCBidXQgSSBkb24ndCAKa25vdyBpZiB0 aGlzIGJ1ZyBjb21lcyBmcm9tIGlzY3NpLXRhcmdldCBvciByYWlkNSBpdHNlbGYuIGlTQ1NJIHRh cmdldCAKaXMgZGlzY29ubmVjdGVkIGJlY2F1c2UgaXN0ZDEgYW5kIG1kX2QwX3JhaWQ1IGtlcm5l bCB0aHJlYWRzIHVzZSAxMDAlIG9mIApDUFUgZWFjaCAhCgpUYXNrczogMjM1IHRvdGFsLCAgIDYg cnVubmluZywgMjI3IHNsZWVwaW5nLCAgIDAgc3RvcHBlZCwgICAyIHpvbWJpZQpDcHUocyk6ICAw LjEldXMsIDEyLjUlc3ksICAwLjAlbmksIDg3LjQlaWQsICAwLjAld2EsICAwLjAlaGksICAwLjAl c2ksIAowLjAlc3QKTWVtOiAgIDQxMzkwMzJrIHRvdGFsLCAgIDIxODQyNGsgdXNlZCwgIDM5MjA2 MDhrIGZyZWUsICAgIDEwMTM2ayBidWZmZXJzClN3YXA6ICA3ODE1NTM2ayB0b3RhbCwgICAgICAg IDBrIHVzZWQsICA3ODE1NTM2ayBmcmVlLCAgICA2NDgwOGsgY2FjaGVkCgogICBQSUQgVVNFUiAg ICAgIFBSICBOSSAgVklSVCAgUkVTICBTSFIgUyAlQ1BVICVNRU0gICAgVElNRSsgIENPTU1BTkQg CgogIDU4MjQgcm9vdCAgICAgIDE1ICAtNSAgICAgMCAgICAwICAgIDAgUiAgMTAwICAwLjAgIDEw OjM0LjI1IGlzdGQxIAoKICA1NTk5IHJvb3QgICAgICAxNSAgLTUgICAgIDAgICAgMCAgICAwIFIg IDEwMCAgMC4wICAgNzoyNS40MyAKbWRfZDBfcmFpZDUKCglSZWdhcmRzLAoKCUpLQgoKLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLQpUaGlzIFNGLm5ldCBlbWFpbCBpcyBzcG9uc29yZWQgYnk6IFNwbHVuayBJbmMu ClN0aWxsIGdyZXBwaW5nIHRocm91Z2ggbG9nIGZpbGVzIHRvIGZpbmQgcHJvYmxlbXM/ICBTdG9w LgpOb3cgU2VhcmNoIGxvZyBldmVudHMgYW5kIGNvbmZpZ3VyYXRpb24gZmlsZXMgdXNpbmcgQUpB WCBhbmQgYSBicm93c2VyLgpEb3dubG9hZCB5b3VyIEZSRUUgY29weSBvZiBTcGx1bmsgbm93ID4+ IGh0dHA6Ly9nZXQuc3BsdW5rLmNvbS8KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX18KSXNjc2l0YXJnZXQtZGV2ZWwgbWFpbGluZyBsaXN0CklzY3NpdGFyZ2V0 LWRldmVsQGxpc3RzLnNvdXJjZWZvcmdlLm5ldApodHRwczovL2xpc3RzLnNvdXJjZWZvcmdlLm5l dC9saXN0cy9saXN0aW5mby9pc2NzaXRhcmdldC1kZXZlbAo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?B?QkVSVFJBTkQgSm/Dq2w=?= Date: Fri, 19 Oct 2007 21:04:15 +0000 Subject: Re: [BUG] Raid1/5 over iSCSI trouble Message-Id: <47191BCF.2000908@systella.fr> List-Id: References: <4714BB92.7040701@systella.fr> <47161CE3.80909@systella.fr> <47181CB2.1060602@tmr.com> <471864F8.9010209@systella.fr> <1192809103.30976.11.camel@dwillia2-linux.ch.intel.com> <4718DE66.8000905@tmr.com> <471916B5.6080709@systella.fr> <47191855.4020402@systella.fr> In-Reply-To: <47191855.4020402@systella.fr> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit To: Bill Davidsen Cc: linux-raid@vger.kernel.org, sparclinux@vger.kernel.org, Dan Williams , iscsitarget-devel@lists.sourceforge.net BERTRAND Joël wrote: > BERTRAND Joël wrote: >> Bill Davidsen wrote: >>> Dan Williams wrote: >>>> On Fri, 2007-10-19 at 01:04 -0700, BERTRAND Joël wrote: >>>> >>>>> I run for 12 hours some dd's (read and write in nullio) >>>>> between >>>>> initiator and target without any disconnection. Thus iSCSI code seems >>>>> to >>>>> be robust. Both initiator and target are alone on a single gigabit >>>>> ethernet link (without any switch). I'm investigating... >>>>> >>>> >>>> Can you reproduce on 2.6.22? >>>> >>>> Also, I do not think this is the cause of your failure, but you have >>>> CONFIG_DMA_ENGINE=y in your config. Setting this to 'n' will compile >>>> out the unneeded checks for offload engines in async_memcpy and >>>> async_xor. >>> >>> Given that offload engines are far less tested code, I think this is >>> a very good thing to try! >> >> I'm trying wihtout CONFIG_DMA_ENGINE=y. istd1 only uses 40% of one >> CPU when I rebuild my raid1 array. 1% of this array was now >> resynchronized without any hang. >> >> Root gershwin:[/usr/scripts] > cat /proc/mdstat >> Personalities : [raid1] [raid6] [raid5] [raid4] >> md7 : active raid1 sdi1[2] md_d0p1[0] >> 1464725632 blocks [2/1] [U_] >> [>....................] recovery = 1.0% (15705536/1464725632) >> finish03.9min speed!875K/sec > > Same result... > > connection2:0: iscsi: detected conn error (1011) > > session2: iscsi: session recovery timed out after 120 secs > sd 4:0:0:0: scsi: Device offlined - not ready after error recovery > sd 4:0:0:0: scsi: Device offlined - not ready after error recovery > sd 4:0:0:0: scsi: Device offlined - not ready after error recovery > sd 4:0:0:0: scsi: Device offlined - not ready after error recovery > sd 4:0:0:0: scsi: Device offlined - not ready after error recovery > sd 4:0:0:0: scsi: Device offlined - not ready after error recovery > sd 4:0:0:0: scsi: Device offlined - not ready after error recovery Sorry for this last mail. I have found another mistake, but I don't know if this bug comes from iscsi-target or raid5 itself. iSCSI target is disconnected because istd1 and md_d0_raid5 kernel threads use 100% of CPU each ! Tasks: 235 total, 6 running, 227 sleeping, 0 stopped, 2 zombie Cpu(s): 0.1%us, 12.5%sy, 0.0%ni, 87.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4139032k total, 218424k used, 3920608k free, 10136k buffers Swap: 7815536k total, 0k used, 7815536k free, 64808k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5824 root 15 -5 0 0 0 R 100 0.0 10:34.25 istd1 5599 root 15 -5 0 0 0 R 100 0.0 7:25.43 md_d0_raid5 Regards, JKB