Xen breaking iSCSI

Xen breaking iSCSI

[Niraj Tolia]

I am trying to run an iSCSI initiator in dom0 and use the device in
dom1. The initiator in question is the Cisco one (3.4.3). For some
reason, running the initiator within Xen causes it to repeatedly
disconnect from Target when one tries to read data from the iSCSI disk
in dom1.

I used two different targets to make sure it was an initiator side
problem. Also, running the initiator with vanilla linux or even just
dom0 causes no problem. Using a hardware initiator (Intel Pro 1000/T)
causes no problems either.

Any idea what the reason could be? This used to work earlier  and
definitely before the s/g changes were merged in. Could this be the
cause?

Niraj


-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com

Re: Xen breaking iSCSI

[Niraj Tolia]

Was trying to beat up on this a little more. Got this OOPS.

ksymoops 2.4.9 on i686 2.4.22-1.2197.nptl.  Options used
     -v xen/experimental/linux-2.4.26-xen0/vmlinux (specified)
     -K (specified)
     -L (specified)
     -O (specified)
     -m xen/experimental/install/boot/System.map-2.4.26-xen0 (specified)

Unable to handle kernel paging request at virtual address 366b2553
c012f9a0
Oops: 0000
CPU:    0
EIP:    0819:[<c012f9a0>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00213207
eax: 00000000   ebx: 366b253b   ecx: 000001d0   edx: 00000010
esi: 00000000   edi: c5eadf80   ebp: c103f550   esp: c119df0c
ds: 0821   es: 0821   ss: 0821
Process kswapd (pid: 4, stackpage=c119d000)<1>
Stack: 00000000 c47c4f80 c5eadf80 c103f550 c5eadf80 c012fafe c5eadf80 00000000 
       c103f550 c03a1df0 00000fe1 c0123213 c103f550 000001d0 00000c80 000001d0 
       0000001c 00000020 000001d0 c03a1df0 c03a1df0 c012345d c119df84 000001d0 
Call Trace: [<c012fafe>] [<c0123213>] [<c012345d>] [<c01234e2>] [<c0123696>] 
   [<c0123708>] [<c0123848>] [<c01237b0>] [<c01afa2e>] [<c01237b0>] 
Warning (Oops_read): Code line not seen, dumping what data is available


>>EIP; c012f9a0 <sync_page_buffers+20/b0>   <=====

>>ebx; 366b253b <__start___xen_guest+366aba41/c00f9506>

Trace; c012fafe <try_to_free_buffers+ce/f0>
Trace; c0123213 <shrink_cache+2d3/380>
Trace; c012345d <shrink_caches+3d/60>
Trace; c01234e2 <try_to_free_pages_zone+62/f0>
Trace; c0123696 <kswapd_balance_pgdat+66/b0>
Trace; c0123708 <kswapd_balance+28/40>
Trace; c0123848 <kswapd+98/c0>
Trace; c01237b0 <kswapd+0/c0>
Trace; c01afa2e <arch_kernel_thread+2e/40>
Trace; c01237b0 <kswapd+0/c0>


1 warning issued.  Results may not be reliable.


-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com

Re: Xen breaking iSCSI

[Niraj Tolia]

Since sending the below email, I started looking at the actual iSCSI
packets with Ethereal. For a read only experiment, it turns out the
disconnect only happens when dom1 requests a read of 16 KB or higher.
4 KB read requests go through just fine.

If only dom0 is performing the reads (with no other domain started),
the request length is irrelevant and all reads will complete
successfully without disconnection.

Any ideas?

Niraj


On Mon, 9 Aug 2004 14:00:34 +0100, Niraj Tolia <ntolia@gmail.com> wrote:
> I am trying to run an iSCSI initiator in dom0 and use the device in
> dom1. The initiator in question is the Cisco one (3.4.3). For some
> reason, running the initiator within Xen causes it to repeatedly
> disconnect from Target when one tries to read data from the iSCSI disk
> in dom1.
> 
> I used two different targets to make sure it was an initiator side
> problem. Also, running the initiator with vanilla linux or even just
> dom0 causes no problem. Using a hardware initiator (Intel Pro 1000/T)
> causes no problems either.
> 
> Any idea what the reason could be? This used to work earlier  and
> definitely before the s/g changes were merged in. Could this be the
> cause?
> 
> Niraj
>


-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com

Re: Re: Xen breaking iSCSI

[Keir Fraser]

Well, I guess it sounds like a merging problem, so it could be that or
a bug in the backend driver that crept in while merging with the 2.6
driver. 

We'll need to be able to reproduce it ourselves, or get more
information by other means. e.g., see if you can track down the bug by
addign tracing to the drivers, or try a repository from a week or two
ago and see if it has the same problem.

 -- Keir

> Since sending the below email, I started looking at the actual iSCSI
> packets with Ethereal. For a read only experiment, it turns out the
> disconnect only happens when dom1 requests a read of 16 KB or higher.
> 4 KB read requests go through just fine.
> 
> If only dom0 is performing the reads (with no other domain started),
> the request length is irrelevant and all reads will complete
> successfully without disconnection.
> 
> Any ideas?
> 
> Niraj
> 
> 
> On Mon, 9 Aug 2004 14:00:34 +0100, Niraj Tolia <ntolia@gmail.com> wrote:
> > I am trying to run an iSCSI initiator in dom0 and use the device in
> > dom1. The initiator in question is the Cisco one (3.4.3). For some
> > reason, running the initiator within Xen causes it to repeatedly
> > disconnect from Target when one tries to read data from the iSCSI disk
> > in dom1.
> > 
> > I used two different targets to make sure it was an initiator side
> > problem. Also, running the initiator with vanilla linux or even just
> > dom0 causes no problem. Using a hardware initiator (Intel Pro 1000/T)
> > causes no problems either.
> > 
> > Any idea what the reason could be? This used to work earlier  and
> > definitely before the s/g changes were merged in. Could this be the
> > cause?
> > 
> > Niraj
> >
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> one more big change to announce. We are now OSTG- Open Source Technology
> Group. Come see the changes on the new OSTG site. www.ostg.com
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com

Re: Re: Xen breaking iSCSI

[Niraj Tolia]

On Mon, 09 Aug 2004 15:14:56 +0100, Keir Fraser
<keir.fraser@cl.cam.ac.uk> wrote:
> 
> Well, I guess it sounds like a merging problem, so it could be that or
> a bug in the backend driver that crept in while merging with the 2.6
> driver.
> 
> We'll need to be able to reproduce it ourselves, or get more
> information by other means. e.g., see if you can track down the bug by
> addign tracing to the drivers, or try a repository from a week or two
> ago and see if it has the same problem.
> 

Traced the bug down to a single changeset. It occurs with this
changeset and after but not before it. I should note that I am using a
xen0 kernel in dom.

ChangeSet@1.1108.17.1, 2004-07-26 16:09:53+01:00, kaf24@scramble.cl.cam.ac.uk
  A better fix for blkdev request merging. Should work for both IDE and
  SCSI, and do as much merging as possible, and also no need for PIO
  fallback mode. 

Let me know if you need any other information.

Thanks,
Niraj

>  -- Keir
> 
> 
> 
> > Since sending the below email, I started looking at the actual iSCSI
> > packets with Ethereal. For a read only experiment, it turns out the
> > disconnect only happens when dom1 requests a read of 16 KB or higher.
> > 4 KB read requests go through just fine.
> >
> > If only dom0 is performing the reads (with no other domain started),
> > the request length is irrelevant and all reads will complete
> > successfully without disconnection.
> >
> > Any ideas?
> >
> > Niraj
> >
> >
> > On Mon, 9 Aug 2004 14:00:34 +0100, Niraj Tolia <ntolia@gmail.com> wrote:
> > > I am trying to run an iSCSI initiator in dom0 and use the device in
> > > dom1. The initiator in question is the Cisco one (3.4.3). For some
> > > reason, running the initiator within Xen causes it to repeatedly
> > > disconnect from Target when one tries to read data from the iSCSI disk
> > > in dom1.
> > >
> > > I used two different targets to make sure it was an initiator side
> > > problem. Also, running the initiator with vanilla linux or even just
> > > dom0 causes no problem. Using a hardware initiator (Intel Pro 1000/T)
> > > causes no problems either.
> > >
> > > Any idea what the reason could be? This used to work earlier  and
> > > definitely before the s/g changes were merged in. Could this be the
> > > cause?
> > >
> > > Niraj
> > >
> >
> > 
> > -------------------------------------------------------
> > This SF.Net email is sponsored by OSTG. Have you noticed the changes on
> > Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
> > one more big change to announce. We are now OSTG- Open Source Technology
> > Group. Come see the changes on the new OSTG site. www.ostg.com
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
>


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: Re: Xen breaking iSCSI

[Ian Pratt]

> On Mon, 09 Aug 2004 15:14:56 +0100, Keir Fraser
> <keir.fraser@cl.cam.ac.uk> wrote:
> > 
> > Well, I guess it sounds like a merging problem, so it could be that or
> > a bug in the backend driver that crept in while merging with the 2.6
> > driver.
> > 
> > We'll need to be able to reproduce it ourselves, or get more
> > information by other means. e.g., see if you can track down the bug by
> > addign tracing to the drivers, or try a repository from a week or two
> > ago and see if it has the same problem.
> > 
> 
> Traced the bug down to a single changeset. It occurs with this
> changeset and after but not before it. I should note that I am using a
> xen0 kernel in dom.
> 
> ChangeSet@1.1108.17.1, 2004-07-26 16:09:53+01:00, kaf24@scramble.cl.cam.ac.uk
>   A better fix for blkdev request merging. Should work for both IDE and
>   SCSI, and do as much merging as possible, and also no need for PIO
>   fallback mode. 
> 
> Let me know if you need any other information.

I suspect this will turn out to be a bug in the iscsi code...

Can you access the iscsi block device within domain 0 OK? Is it
only when you export it to another domain that there's a problem?

Have you tried using the 2.6 kernel and iscsi-4.0.1 driver?

Ian


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285

Re: Re: Xen breaking iSCSI

[Niraj Tolia]

On Thu, 12 Aug 2004 11:15:26 +0100, Ian Pratt <ian.pratt@cl.cam.ac.uk> wrote:
> > On Mon, 09 Aug 2004 15:14:56 +0100, Keir Fraser
> > <keir.fraser@cl.cam.ac.uk> wrote:
> > >
> > > Well, I guess it sounds like a merging problem, so it could be that or
> > > a bug in the backend driver that crept in while merging with the 2.6
> > > driver.
> > >
> > > We'll need to be able to reproduce it ourselves, or get more
> > > information by other means. e.g., see if you can track down the bug by
> > > addign tracing to the drivers, or try a repository from a week or two
> > > ago and see if it has the same problem.
> > >
> >
> > Traced the bug down to a single changeset. It occurs with this
> > changeset and after but not before it. I should note that I am using a
> > xen0 kernel in dom.
> >
> > ChangeSet@1.1108.17.1, 2004-07-26 16:09:53+01:00, kaf24@scramble.cl.cam.ac.uk
> >   A better fix for blkdev request merging. Should work for both IDE and
> >   SCSI, and do as much merging as possible, and also no need for PIO
> >   fallback mode.
> >
> > Let me know if you need any other information.
> 
> I suspect this will turn out to be a bug in the iscsi code...
> 
> Can you access the iscsi block device within domain 0 OK? Is it
> only when you export it to another domain that there's a problem?

That is correct. All accesses within dom0 are ok. Its only when its
exported to and used from another domain that I see the problems.

> 
> Have you tried using the 2.6 kernel and iscsi-4.0.1 driver?
> 

I have not done that yet. I will try to give it a shot. I am assuming
you are talking about 2.6 within Xen.

Niraj


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285