From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: Create a iSCSI DomU with disks in another DomU running on the same Dom0 Date: Fri, 21 Dec 2012 12:35:13 -0500 Message-ID: <20121221173513.GB27893@phenom.dumpdata.com> References: <50D41DF3.306@citrix.com> <20121221140320.GD25526@phenom.dumpdata.com> <50D47678.2050903@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: <50D47678.2050903@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Roger Pau =?iso-8859-1?Q?Monn=E9?= Cc: xen-devel List-Id: xen-devel@lists.xenproject.org On Fri, Dec 21, 2012 at 03:47:20PM +0100, Roger Pau Monn=E9 wrote: > On 21/12/12 15:03, Konrad Rzeszutek Wilk wrote: > > On Fri, Dec 21, 2012 at 09:29:39AM +0100, Roger Pau Monn=E9 wrote: > >> Hello, > >> > >> I'm trying to use a strange setup, that consists in having a DomU > >> serving iSCSI targets to the Dom0, that will use this targets as disks > >> for other DomUs. I've tried to set up this iSCSI target DomU using both > >> Debian Squeeze/Wheezy (with kernels 2.6.32 and 3.2) and ISCSI > >> Enterprise Target (IET), and when launching the DomU I get this messag= es > >> from Xen: > >> > >> (XEN) mm.c:1925:d0 Error pfn 157e68: rd=3Dffff83019e60c000, od=3Dffff8= 30141405000, caf=3D8000000000000003, taf=3D7400000000000001 > >> (XEN) Xen WARN at mm.c:1926 > >> (XEN) ----[ Xen-4.3-unstable x86_64 debug=3Dy Not tainted ]---- > >> (XEN) CPU: 0 > >> (XEN) RIP: e008:[] get_page+0xd5/0x101 > >> (XEN) RFLAGS: 0000000000010286 CONTEXT: hypervisor > >> (XEN) rax: 0000000000000000 rbx: ffff830141405000 rcx: 00000000000= 00000 > >> (XEN) rdx: ffff82c480300920 rsi: 000000000000000a rdi: ffff82c4802= 766e8 > >> (XEN) rbp: ffff82c4802bfbf8 rsp: ffff82c4802bfba8 r8: 00000000000= 00004 > >> (XEN) r9: 0000000000000004 r10: 0000000000000004 r11: 00000000000= 00001 > >> (XEN) r12: 0000000000157e68 r13: ffff83019e60c000 r14: 74000000000= 00001 > >> (XEN) r15: 8000000000000003 cr0: 000000008005003b cr4: 00000000000= 026f0 > >> (XEN) cr3: 000000011c180000 cr2: 00007f668d1eb000 > >> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 > >> (XEN) Xen stack trace from rsp=3Dffff82c4802bfba8: > >> (XEN) ffff830141405000 8000000000000003 7400000000000001 0000000000= 145028 > >> (XEN) ffff82f6028a0510 ffff83019e60c000 ffff82f602afcd00 ffff82c480= 2bfd28 > >> (XEN) ffff82c4802bfd18 0000000000157e68 ffff82c4802bfc58 ffff82c480= 109ba3 > >> (XEN) ffffffffffffffff 0000000000000000 ffff83011c977fb8 0000000061= dfc3f0 > >> (XEN) 0000000000000001 ffffffffffff8000 0000000000000002 ffff83011d= 555000 > >> (XEN) ffff83019e60c000 0000000000000000 ffff82c4802bfd98 ffff82c480= 10c607 > >> (XEN) ffff82c4802bfd34 ffff82c4802bfd30 ffff82c400000001 0000000000= 11cf90 > >> (XEN) 0000000000000000 ffff82c4802b8000 ffff82c4802b8000 ffff82c480= 2b8000 > >> (XEN) ffff82c4802b8000 ffff82c4802bfd5c 000000029e60c000 ffff82c480= 300920 > >> (XEN) ffff82c4802b8000 ffff82c4802bfd38 00000005802bfd38 ffff82c480= 2b8000 > >> (XEN) ffff82c400000000 0000000000000001 ffffc90000028b10 ffffc90000= 028b10 > >> (XEN) ffff8300dfb03000 0000000000000000 0000000000000000 0000000000= 145028 > >> (XEN) 000000000011cf7c 0000000000001000 0000000000157e68 0000000000= 007ff0 > >> (XEN) 000000000000027e 000000000042000d 0000000000020b50 ffff8300df= df0000 > >> (XEN) ffff82c4802bfd78 ffffc90000028ac0 ffffc90000028ac0 ffff880185= f6fd58 > >> (XEN) ffff880185f6fd78 0000000000000005 ffff82c4802bfef8 ffff82c480= 10eb65 > >> (XEN) ffff82c4802bfdc8 ffff82c480300960 ffff82c4802bfe18 ffff82c480= 181831 > >> (XEN) 000000000006df66 000032cfdc175ce6 0000000000000000 0000000000= 000000 > >> (XEN) 0000000000000000 0000000000000005 ffff82c4802bfe28 ffff8300df= b03000 > >> (XEN) ffff8300dfdf0000 0000150e11a417f8 0000000000000002 ffff82c480= 300948 > >> (XEN) Xen call trace: > >> (XEN) [] get_page+0xd5/0x101 > >> (XEN) [] __get_paged_frame+0xbf/0x162 > >> (XEN) [] gnttab_copy+0x4c6/0x91a > >> (XEN) [] do_grant_table_op+0x12ad/0x1b23 > >> (XEN) [] syscall_enter+0xeb/0x145 > >> (XEN) = > >> (XEN) grant_table.c:2076:d0 source frame ffffffffffffffff invalid. > >> (XEN) mm.c:1925:d0 Error pfn 157e68: rd=3Dffff83019e60c000, od=3Dffff8= 30141405000, caf=3D8000000000000003, taf=3D7400000000000001 > >> (XEN) Xen WARN at mm.c:1926 > >> (XEN) ----[ Xen-4.3-unstable x86_64 debug=3Dy Not tainted ]---- > >> (XEN) CPU: 0 > >> (XEN) RIP: e008:[] get_page+0xd5/0x101 > >> (XEN) RFLAGS: 0000000000010286 CONTEXT: hypervisor > >> (XEN) rax: 0000000000000000 rbx: ffff830141405000 rcx: 00000000000= 00000 > >> (XEN) rdx: ffff82c480300920 rsi: 000000000000000a rdi: ffff82c4802= 766e8 > >> (XEN) rbp: ffff82c4802bfbf8 rsp: ffff82c4802bfba8 r8: 00000000000= 00004 > >> (XEN) r9: 0000000000000004 r10: 0000000000000004 r11: 00000000000= 00001 > >> (XEN) r12: 0000000000157e68 r13: ffff83019e60c000 r14: 74000000000= 00001 > >> (XEN) r15: 8000000000000003 cr0: 000000008005003b cr4: 00000000000= 026f0 > >> (XEN) cr3: 000000011c180000 cr2: 00007f668d1eb000 > >> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 > >> (XEN) Xen stack trace from rsp=3Dffff82c4802bfba8: > >> (XEN) ffff830141405000 8000000000000003 7400000000000001 0000000000= 14581d > >> (XEN) ffff82f6028b03b0 ffff83019e60c000 ffff82f602afcd00 ffff82c480= 2bfd28 > >> (XEN) ffff82c4802bfd18 0000000000157e68 ffff82c4802bfc58 ffff82c480= 109ba3 > >> (XEN) ffffffffffffffff 0000000000000000 ffff83011c977fb8 0000000061= dfc308 > >> (XEN) 0000000000000000 ffffffffffff8000 0000000000000001 ffff83011d= 555000 > >> (XEN) ffff83019e60c000 0000000000000000 ffff82c4802bfd98 ffff82c480= 10c607 > >> (XEN) ffff82c4802bfd34 ffff82c4802bfd30 ffff82c400000001 0000000000= 11cf90 > >> (XEN) 0000000000000000 ffff82c4802b8000 ffff82c4802b8000 ffff82c480= 2b8000 > >> (XEN) ffff82c4802b8000 ffff82c4802bfd5c 000000029e60c000 ffff82c480= 300920 > >> (XEN) ffff82c4802b8000 ffff82c4802bfd38 00000002802bfd38 ffff82c480= 2b8000 > >> (XEN) ffffffff00000000 0000000000000001 ffffc90000028b60 ffffc90000= 028b60 > >> (XEN) ffff8300dfb03000 0000000000000000 0000000000000000 0000000000= 14581d > >> (XEN) 00000000000deb3e 0000000000001000 0000000000157e68 000000000b= 507ff0 > >> (XEN) 0000000000000261 000000000042000d 00000000000204b0 ffffc90000= 028b38 > >> (XEN) 0000000000000002 ffffc90000028b38 ffffc90000028b38 ffff880185= f6fd58 > >> (XEN) ffff880185f6fd78 0000000000000005 ffff82c4802bfef8 ffff82c480= 10eb65 > >> (XEN) ffff82c4802bfdc8 ffff82c480300960 ffff82c4802bfe18 ffff82c480= 181831 > >> (XEN) 000000000006df66 000032cfdc175ce6 0000000000000000 0000000000= 000000 > >> (XEN) 0000000000000000 0000000000000005 ffff82c4802bfe28 0000000000= 000086 > >> (XEN) ffff82c4802bfe28 ffff82c480125eae ffff83019e60c000 0000000000= 000286 > >> (XEN) Xen call trace: > >> (XEN) [] get_page+0xd5/0x101 > >> (XEN) [] __get_paged_frame+0xbf/0x162 > >> (XEN) [] gnttab_copy+0x4c6/0x91a > >> (XEN) [] do_grant_table_op+0x12ad/0x1b23 > >> (XEN) [] syscall_enter+0xeb/0x145 > >> (XEN) = > >> (XEN) grant_table.c:2076:d0 source frame ffffffffffffffff invalid. > >> > >> (Note that I've added a WARN() to mm.c:1925 to see where the > >> get_page call was coming from). > >> > >> Connecting the iSCSI disks to another Dom0 works fine, so this > >> problem only happens when trying to connect the disks to the > >> Dom0 where the DomU is running. > > = > > Is this happening when the 'disks' are exported to the domUs? > > Are they exported via QEMU or xen-blkback? > = > The iSCSI disks are connected to the DomUs using blkback, and this is > happening when the DomU tries to access it's disks. > = > >> > >> I've replaced the Linux DomU serving iSCSI targets with a > >> NetBSD DomU, and the problems disappears, and I'm able to > >> attach the targets shared by the DomU to the Dom0 without > >> issues. > >> > >> The problem seems to come from netfront/netback, does anyone > >> have a clue about what might cause this bad interaction > >> between IET and netfront/netback? > > = > > Or it might be that we are re-using the PFN for blkback/blkfront > > and using the m2p overrides and overwritting the netfront/netback > > m2p overrides? > = > What's strange is that this doesn't happen when the domain that has the > targets is a NetBSD PV. There are also problems when blkback is not used > (see below), so I guess the problem is between netfront/netback and IET. > = > > Is this with an HVM domU or PV domU? > = > Both domains (the domain holding the iSCSI targets, and the created > guests) are PV. > = > Also, I've forgot to say that in the previous email, but if I just > connect the iSCSI disks to the Dom0, I don't see any errors from Xen, > but the Dom0 kernel starts complaining: > = > [70272.569607] sd 14:0:0:0: [sdc] > [70272.569611] Sense Key : Medium Error [current] > [70272.569619] Info fld=3D0x0 > [70272.569623] sd 14:0:0:0: [sdc] > [70272.569627] Add. Sense: Unrecovered read error > [70272.569633] sd 14:0:0:0: [sdc] CDB: > [70272.569637] Read(10): 28 00 00 00 00 00 00 00 08 00 > [70272.569662] end_request: critical target error, dev sdc, sector 0 > [70277.571208] sd 14:0:0:0: [sdc] Unhandled sense code > [70277.571220] sd 14:0:0:0: [sdc] > [70277.571224] Result: hostbyte=3DDID_OK driverbyte=3DDRIVER_SENSE > [70277.571229] sd 14:0:0:0: [sdc] > [70277.571233] Sense Key : Medium Error [current] > [70277.571241] Info fld=3D0x0 > [70277.571245] sd 14:0:0:0: [sdc] > [70277.571249] Add. Sense: Unrecovered read error > [70277.571255] sd 14:0:0:0: [sdc] CDB: > [70277.571259] Read(10): 28 00 00 00 00 00 00 00 08 00 > [70277.571284] end_request: critical target error, dev sdc, sector 0 > [70282.572768] sd 14:0:0:0: [sdc] Unhandled sense code > [70282.572781] sd 14:0:0:0: [sdc] > [70282.572785] Result: hostbyte=3DDID_OK driverbyte=3DDRIVER_SENSE > [70282.572790] sd 14:0:0:0: [sdc] > [70282.572794] Sense Key : Medium Error [current] > [70282.572802] Info fld=3D0x0 > [70282.572806] sd 14:0:0:0: [sdc] > [70282.572810] Add. Sense: Unrecovered read error > [70282.572816] sd 14:0:0:0: [sdc] CDB: > [70282.572820] Read(10): 28 00 00 00 00 00 00 00 08 00 > [70282.572846] end_request: critical target error, dev sdc, sector 0 > [70287.574397] sd 14:0:0:0: [sdc] Unhandled sense code > [70287.574409] sd 14:0:0:0: [sdc] > [70287.574413] Result: hostbyte=3DDID_OK driverbyte=3DDRIVER_SENSE > [70287.574418] sd 14:0:0:0: [sdc] > [70287.574422] Sense Key : Medium Error [current] > [70287.574430] Info fld=3D0x0 > [70287.574434] sd 14:0:0:0: [sdc] > [70287.574438] Add. Sense: Unrecovered read error > [70287.574445] sd 14:0:0:0: [sdc] CDB: > [70287.574448] Read(10): 28 00 00 00 00 00 00 00 08 00 > [70287.574474] end_request: critical target error, dev sdc, sector 0 > = > When I try to attach the targets to another Dom0, everything works fine, > the problem only happens when the iSCSI target is a DomU and you attach > the disks from the Dom0 on the same machine. I think we are just swizzling the PFNs with a different MFN when you do the domU -> domX, using two ring protocols. Weird thought as the m2p code has checks WARN_ON(PagePrivate(..)) to catch this sort of thing. What happens if the dom0/domU are all 3.8 with the persistent grant patches? > = > Thanks, Roger. > =