Trying to track down odd behavior, errors in xen dmesg

Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

Hi,

I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
I am importing a iSCSI lun into dom0 from a storage array (using
open-iscsi), this is used as the disk [phy:] for my domU.
If I use HVM, the install works correctly. As soon as I switch to pv
for the disk drivers (after installing a xen kernel in the domU) I get
a slew of messages in xm dmesg:

(XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
(XEN) printk: 1851 messages suppressed.
(XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
(XEN) printk: 1354 messages suppressed.
(XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
(XEN) printk: 1890 messages suppressed.
(XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
(XEN) printk: 2153 messages suppressed.
(XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
(XEN) printk: 1912 messages suppressed.
(XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
(XEN) printk: 2287 messages suppressed.
(XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
Disk speed is very slow as well. Running DD in the domU to write a
file to disk causes more error messages to appear.  It also badly
corrupts the disk in some cases.
Testing from a local LVM works without error in all cases.

Any suggestions?
--Keith

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
> Hi,
> 
> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
> I am importing a iSCSI lun into dom0 from a storage array (using
> open-iscsi), this is used as the disk [phy:] for my domU.

> If I use HVM, the install works correctly. As soon as I switch to pv
> for the disk drivers (after installing a xen kernel in the domU) I get
> a slew of messages in xm dmesg:
This problem does not occur if the phy is pointing to a local disk?
> 
> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
> (XEN) printk: 1851 messages suppressed.
> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
> (XEN) printk: 1354 messages suppressed.
> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
> (XEN) printk: 1890 messages suppressed.
> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
> (XEN) printk: 2153 messages suppressed.
> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
> (XEN) printk: 1912 messages suppressed.
> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
> (XEN) printk: 2287 messages suppressed.
> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
> Disk speed is very slow as well. Running DD in the domU to write a
> file to disk causes more error messages to appear.  It also badly
> corrupts the disk in some cases.
> Testing from a local LVM works without error in all cases.
> 
> Any suggestions?
> --Keith
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

On Thu, Apr 7, 2011 at 8:54 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
>> Hi,
>>
>> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
>> I am importing a iSCSI lun into dom0 from a storage array (using
>> open-iscsi), this is used as the disk [phy:] for my domU.
>
>> If I use HVM, the install works correctly. As soon as I switch to pv
>> for the disk drivers (after installing a xen kernel in the domU) I get
>> a slew of messages in xm dmesg:
> This problem does not occur if the phy is pointing to a local disk?

No issues when pointing to local disk, partition or LVM.

>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
>> (XEN) printk: 1851 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
>> (XEN) printk: 1354 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
>> (XEN) printk: 1890 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
>> (XEN) printk: 2153 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
>> (XEN) printk: 1912 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
>> (XEN) printk: 2287 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
>> Disk speed is very slow as well. Running DD in the domU to write a
>> file to disk causes more error messages to appear.  It also badly
>> corrupts the disk in some cases.
>> Testing from a local LVM works without error in all cases.
>>
>> Any suggestions?
>> --Keith
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

A bit more info to add, I just tested with AOE (ata over ethernet)
instead of iSCSI and I see the same issue.

--Keith

On Thu, Apr 7, 2011 at 8:54 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
>> Hi,
>>
>> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
>> I am importing a iSCSI lun into dom0 from a storage array (using
>> open-iscsi), this is used as the disk [phy:] for my domU.
>
>> If I use HVM, the install works correctly. As soon as I switch to pv
>> for the disk drivers (after installing a xen kernel in the domU) I get
>> a slew of messages in xm dmesg:
> This problem does not occur if the phy is pointing to a local disk?
>>
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
>> (XEN) printk: 1851 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
>> (XEN) printk: 1354 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
>> (XEN) printk: 1890 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
>> (XEN) printk: 2153 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
>> (XEN) printk: 1912 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
>> (XEN) printk: 2287 messages suppressed.
>> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
>> Disk speed is very slow as well. Running DD in the domU to write a
>> file to disk causes more error messages to appear.  It also badly
>> corrupts the disk in some cases.
>> Testing from a local LVM works without error in all cases.
>>
>> Any suggestions?
>> --Keith
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

On Thu, Apr 07, 2011 at 10:52:37AM -0500, Keith Swett wrote:
> A bit more info to add, I just tested with AOE (ata over ethernet)
> instead of iSCSI and I see the same issue.

Sure, but does it occur if you devices that are not using the network.
Like a local disk?

> 
> --Keith
> 
> On Thu, Apr 7, 2011 at 8:54 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
> >> Hi,
> >>
> >> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
> >> I am importing a iSCSI lun into dom0 from a storage array (using
> >> open-iscsi), this is used as the disk [phy:] for my domU.
> >
> >> If I use HVM, the install works correctly. As soon as I switch to pv
> >> for the disk drivers (after installing a xen kernel in the domU) I get
> >> a slew of messages in xm dmesg:
> > This problem does not occur if the phy is pointing to a local disk?
> >>
> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
> >> (XEN) printk: 1851 messages suppressed.
> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
> >> (XEN) printk: 1354 messages suppressed.
> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
> >> (XEN) printk: 1890 messages suppressed.
> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
> >> (XEN) printk: 2153 messages suppressed.
> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
> >> (XEN) printk: 1912 messages suppressed.
> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
> >> (XEN) printk: 2287 messages suppressed.
> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
> >> Disk speed is very slow as well. Running DD in the domU to write a
> >> file to disk causes more error messages to appear.  It also badly
> >> corrupts the disk in some cases.
> >> Testing from a local LVM works without error in all cases.
> >>
> >> Any suggestions?
> >> --Keith
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

Local disk, partition, and LVM all work correctly.

--Keith

On Thu, Apr 7, 2011 at 11:01 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Thu, Apr 07, 2011 at 10:52:37AM -0500, Keith Swett wrote:
>> A bit more info to add, I just tested with AOE (ata over ethernet)
>> instead of iSCSI and I see the same issue.
>
> Sure, but does it occur if you devices that are not using the network.
> Like a local disk?
>
>>
>> --Keith
>>
>> On Thu, Apr 7, 2011 at 8:54 AM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
>> >> Hi,
>> >>
>> >> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
>> >> I am importing a iSCSI lun into dom0 from a storage array (using
>> >> open-iscsi), this is used as the disk [phy:] for my domU.
>> >
>> >> If I use HVM, the install works correctly. As soon as I switch to pv
>> >> for the disk drivers (after installing a xen kernel in the domU) I get
>> >> a slew of messages in xm dmesg:
>> > This problem does not occur if the phy is pointing to a local disk?
>> >>
>> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
>> >> (XEN) printk: 1851 messages suppressed.
>> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
>> >> (XEN) printk: 1354 messages suppressed.
>> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
>> >> (XEN) printk: 1890 messages suppressed.
>> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
>> >> (XEN) printk: 2153 messages suppressed.
>> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
>> >> (XEN) printk: 1912 messages suppressed.
>> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
>> >> (XEN) printk: 2287 messages suppressed.
>> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
>> >> Disk speed is very slow as well. Running DD in the domU to write a
>> >> file to disk causes more error messages to appear.  It also badly
>> >> corrupts the disk in some cases.
>> >> Testing from a local LVM works without error in all cases.
>> >>
>> >> Any suggestions?
>> >> --Keith
>> >>
>> >> _______________________________________________
>> >> Xen-devel mailing list
>> >> Xen-devel@lists.xensource.com
>> >> http://lists.xensource.com/xen-devel
>> >
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

On Thu, Apr 07, 2011 at 11:02:55AM -0500, Keith Swett wrote:
> Local disk, partition, and LVM all work correctly.

And you mentioned that at the end and I completly missed it. Ooops.
Sorry about that.

You mentioned Xen 4.0.1... so when did this work? What kernel/hypervisor
did not exhibit this problem?


> 
> --Keith
> 
> On Thu, Apr 7, 2011 at 11:01 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Thu, Apr 07, 2011 at 10:52:37AM -0500, Keith Swett wrote:
> >> A bit more info to add, I just tested with AOE (ata over ethernet)
> >> instead of iSCSI and I see the same issue.
> >
> > Sure, but does it occur if you devices that are not using the network.
> > Like a local disk?
> >
> >>
> >> --Keith
> >>
> >> On Thu, Apr 7, 2011 at 8:54 AM, Konrad Rzeszutek Wilk
> >> <konrad.wilk@oracle.com> wrote:
> >> > On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
> >> >> Hi,
> >> >>
> >> >> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
> >> >> I am importing a iSCSI lun into dom0 from a storage array (using
> >> >> open-iscsi), this is used as the disk [phy:] for my domU.
> >> >
> >> >> If I use HVM, the install works correctly. As soon as I switch to pv
> >> >> for the disk drivers (after installing a xen kernel in the domU) I get
> >> >> a slew of messages in xm dmesg:
> >> > This problem does not occur if the phy is pointing to a local disk?
> >> >>
> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
> >> >> (XEN) printk: 1851 messages suppressed.
> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
> >> >> (XEN) printk: 1354 messages suppressed.
> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
> >> >> (XEN) printk: 1890 messages suppressed.
> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
> >> >> (XEN) printk: 2153 messages suppressed.
> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
> >> >> (XEN) printk: 1912 messages suppressed.
> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
> >> >> (XEN) printk: 2287 messages suppressed.
> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
> >> >> Disk speed is very slow as well. Running DD in the domU to write a
> >> >> file to disk causes more error messages to appear.  It also badly
> >> >> corrupts the disk in some cases.
> >> >> Testing from a local LVM works without error in all cases.
> >> >>
> >> >> Any suggestions?
> >> >> --Keith
> >> >>
> >> >> _______________________________________________
> >> >> Xen-devel mailing list
> >> >> Xen-devel@lists.xensource.com
> >> >> http://lists.xensource.com/xen-devel
> >> >
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

Here's a summary:

All installs are Debian Squeeze

These stay the same for all tests:
Host is Xen 4.0.1-2 (AMD64 install)
Dom0 Kernel is 2.6.32-5-xen-amd64

Test procedure:
Install Debian Distro via PXE boot using HVM.
If successful, install xen version of kernel, reboot in non-HVM mode.

DomUs:

with iSCSI:
debian pxe netboot installer i386 in HVM mode WORKS (disk detects as
Qemu hard drive)
FAIL after installing a xen kernel (linux-image-xen-686_2.6.32+29) and
switching to builder=linux

debian pxe netboot installer AMD64 in HVM mode FAIL (disk detects as
Xen Virtual Hard drive)

with AOE:
debian pxe netboot installer AMD64 in HVM mode FAIL (disk detects as
Xen Virtual Hard drive)

with LVM:
debian pxe netboot installer i386 in HVM mode WORKS (disk detects as
Qemu hard drive)
WORKS after installing a xen kernel (linux-image-xen-686_2.6.32+29)
and switching to builder=linux

debian pxe netboot installer AMD64 in HVM mode WORKS (disk detects as
Xen Virtual hard drive)
WORKS after installing a xen kernel
(linux-image-2.6-xen-amd64_2.6.32+29) and switching to builder=linux

the tests for physical drives are identical to the LVM tests.

In all FAIL cases, similar messages appear in xm dmesg.

--Keith



On Thu, Apr 7, 2011 at 11:23 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Thu, Apr 07, 2011 at 11:02:55AM -0500, Keith Swett wrote:
>> Local disk, partition, and LVM all work correctly.
>
> And you mentioned that at the end and I completly missed it. Ooops.
> Sorry about that.
>
> You mentioned Xen 4.0.1... so when did this work? What kernel/hypervisor
> did not exhibit this problem?
>
>
>>
>> --Keith
>>
>> On Thu, Apr 7, 2011 at 11:01 AM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Thu, Apr 07, 2011 at 10:52:37AM -0500, Keith Swett wrote:
>> >> A bit more info to add, I just tested with AOE (ata over ethernet)
>> >> instead of iSCSI and I see the same issue.
>> >
>> > Sure, but does it occur if you devices that are not using the network.
>> > Like a local disk?
>> >
>> >>
>> >> --Keith
>> >>
>> >> On Thu, Apr 7, 2011 at 8:54 AM, Konrad Rzeszutek Wilk
>> >> <konrad.wilk@oracle.com> wrote:
>> >> > On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
>> >> >> Hi,
>> >> >>
>> >> >> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
>> >> >> I am importing a iSCSI lun into dom0 from a storage array (using
>> >> >> open-iscsi), this is used as the disk [phy:] for my domU.
>> >> >
>> >> >> If I use HVM, the install works correctly. As soon as I switch to pv
>> >> >> for the disk drivers (after installing a xen kernel in the domU) I get
>> >> >> a slew of messages in xm dmesg:
>> >> > This problem does not occur if the phy is pointing to a local disk?
>> >> >>
>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
>> >> >> (XEN) printk: 1851 messages suppressed.
>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
>> >> >> (XEN) printk: 1354 messages suppressed.
>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
>> >> >> (XEN) printk: 1890 messages suppressed.
>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
>> >> >> (XEN) printk: 2153 messages suppressed.
>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
>> >> >> (XEN) printk: 1912 messages suppressed.
>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
>> >> >> (XEN) printk: 2287 messages suppressed.
>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
>> >> >> Disk speed is very slow as well. Running DD in the domU to write a
>> >> >> file to disk causes more error messages to appear.  It also badly
>> >> >> corrupts the disk in some cases.
>> >> >> Testing from a local LVM works without error in all cases.
>> >> >>
>> >> >> Any suggestions?
>> >> >> --Keith
>> >> >>
>> >> >> _______________________________________________
>> >> >> Xen-devel mailing list
>> >> >> Xen-devel@lists.xensource.com
>> >> >> http://lists.xensource.com/xen-devel
>> >> >
>> >>
>> >> _______________________________________________
>> >> Xen-devel mailing list
>> >> Xen-devel@lists.xensource.com
>> >> http://lists.xensource.com/xen-devel
>> >
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

Another update:

Retested using Xen 4.0.1-2 (i386 install)
and Dom0 Kernel 2.6.32-5-xen-686
with the same kernel for the domU.  The same strange behavior still
shows up when using iSCSI backing.

--Keith

On Thu, Apr 7, 2011 at 11:39 AM, Keith Swett <swettk@gmail.com> wrote:
> Here's a summary:
>
> All installs are Debian Squeeze
>
> These stay the same for all tests:
> Host is Xen 4.0.1-2 (AMD64 install)
> Dom0 Kernel is 2.6.32-5-xen-amd64
>
> Test procedure:
> Install Debian Distro via PXE boot using HVM.
> If successful, install xen version of kernel, reboot in non-HVM mode.
>
> DomUs:
>
> with iSCSI:
> debian pxe netboot installer i386 in HVM mode WORKS (disk detects as
> Qemu hard drive)
> FAIL after installing a xen kernel (linux-image-xen-686_2.6.32+29) and
> switching to builder=linux
>
> debian pxe netboot installer AMD64 in HVM mode FAIL (disk detects as
> Xen Virtual Hard drive)
>
> with AOE:
> debian pxe netboot installer AMD64 in HVM mode FAIL (disk detects as
> Xen Virtual Hard drive)
>
> with LVM:
> debian pxe netboot installer i386 in HVM mode WORKS (disk detects as
> Qemu hard drive)
> WORKS after installing a xen kernel (linux-image-xen-686_2.6.32+29)
> and switching to builder=linux
>
> debian pxe netboot installer AMD64 in HVM mode WORKS (disk detects as
> Xen Virtual hard drive)
> WORKS after installing a xen kernel
> (linux-image-2.6-xen-amd64_2.6.32+29) and switching to builder=linux
>
> the tests for physical drives are identical to the LVM tests.
>
> In all FAIL cases, similar messages appear in xm dmesg.
>
> --Keith
>
>
>
> On Thu, Apr 7, 2011 at 11:23 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
>> On Thu, Apr 07, 2011 at 11:02:55AM -0500, Keith Swett wrote:
>>> Local disk, partition, and LVM all work correctly.
>>
>> And you mentioned that at the end and I completly missed it. Ooops.
>> Sorry about that.
>>
>> You mentioned Xen 4.0.1... so when did this work? What kernel/hypervisor
>> did not exhibit this problem?
>>
>>
>>>
>>> --Keith
>>>
>>> On Thu, Apr 7, 2011 at 11:01 AM, Konrad Rzeszutek Wilk
>>> <konrad.wilk@oracle.com> wrote:
>>> > On Thu, Apr 07, 2011 at 10:52:37AM -0500, Keith Swett wrote:
>>> >> A bit more info to add, I just tested with AOE (ata over ethernet)
>>> >> instead of iSCSI and I see the same issue.
>>> >
>>> > Sure, but does it occur if you devices that are not using the network.
>>> > Like a local disk?
>>> >
>>> >>
>>> >> --Keith
>>> >>
>>> >> On Thu, Apr 7, 2011 at 8:54 AM, Konrad Rzeszutek Wilk
>>> >> <konrad.wilk@oracle.com> wrote:
>>> >> > On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
>>> >> >> Hi,
>>> >> >>
>>> >> >> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
>>> >> >> I am importing a iSCSI lun into dom0 from a storage array (using
>>> >> >> open-iscsi), this is used as the disk [phy:] for my domU.
>>> >> >
>>> >> >> If I use HVM, the install works correctly. As soon as I switch to pv
>>> >> >> for the disk drivers (after installing a xen kernel in the domU) I get
>>> >> >> a slew of messages in xm dmesg:
>>> >> > This problem does not occur if the phy is pointing to a local disk?
>>> >> >>
>>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
>>> >> >> (XEN) printk: 1851 messages suppressed.
>>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
>>> >> >> (XEN) printk: 1354 messages suppressed.
>>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
>>> >> >> (XEN) printk: 1890 messages suppressed.
>>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
>>> >> >> (XEN) printk: 2153 messages suppressed.
>>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
>>> >> >> (XEN) printk: 1912 messages suppressed.
>>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
>>> >> >> (XEN) printk: 2287 messages suppressed.
>>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
>>> >> >> Disk speed is very slow as well. Running DD in the domU to write a
>>> >> >> file to disk causes more error messages to appear.  It also badly
>>> >> >> corrupts the disk in some cases.
>>> >> >> Testing from a local LVM works without error in all cases.
>>> >> >>
>>> >> >> Any suggestions?
>>> >> >> --Keith
>>> >> >>
>>> >> >> _______________________________________________
>>> >> >> Xen-devel mailing list
>>> >> >> Xen-devel@lists.xensource.com
>>> >> >> http://lists.xensource.com/xen-devel
>>> >> >
>>> >>
>>> >> _______________________________________________
>>> >> Xen-devel mailing list
>>> >> Xen-devel@lists.xensource.com
>>> >> http://lists.xensource.com/xen-devel
>>> >
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>
>

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

On Fri, Apr 08, 2011 at 08:40:10AM -0500, Keith Swett wrote:
> Another update:
> 
> Retested using Xen 4.0.1-2 (i386 install)
> and Dom0 Kernel 2.6.32-5-xen-686
> with the same kernel for the domU.  The same strange behavior still
> shows up when using iSCSI backing.

So way back at VirtualIron we had some of these issues (if this 
is the same issue) and we fixed it by holding on the pages that
went to the network stack and took a bit of time to get back.

I was going to dust them of next week to prepare the blkback for
upstream submission. Are you up for being a beta-tester? Will
require some kernel recompilation.

> 
> --Keith
> 
> On Thu, Apr 7, 2011 at 11:39 AM, Keith Swett <swettk@gmail.com> wrote:
> > Here's a summary:
> >
> > All installs are Debian Squeeze
> >
> > These stay the same for all tests:
> > Host is Xen 4.0.1-2 (AMD64 install)
> > Dom0 Kernel is 2.6.32-5-xen-amd64
> >
> > Test procedure:
> > Install Debian Distro via PXE boot using HVM.
> > If successful, install xen version of kernel, reboot in non-HVM mode.
> >
> > DomUs:
> >
> > with iSCSI:
> > debian pxe netboot installer i386 in HVM mode WORKS (disk detects as
> > Qemu hard drive)
> > FAIL after installing a xen kernel (linux-image-xen-686_2.6.32+29) and
> > switching to builder=linux
> >
> > debian pxe netboot installer AMD64 in HVM mode FAIL (disk detects as
> > Xen Virtual Hard drive)
> >
> > with AOE:
> > debian pxe netboot installer AMD64 in HVM mode FAIL (disk detects as
> > Xen Virtual Hard drive)
> >
> > with LVM:
> > debian pxe netboot installer i386 in HVM mode WORKS (disk detects as
> > Qemu hard drive)
> > WORKS after installing a xen kernel (linux-image-xen-686_2.6.32+29)
> > and switching to builder=linux
> >
> > debian pxe netboot installer AMD64 in HVM mode WORKS (disk detects as
> > Xen Virtual hard drive)
> > WORKS after installing a xen kernel
> > (linux-image-2.6-xen-amd64_2.6.32+29) and switching to builder=linux
> >
> > the tests for physical drives are identical to the LVM tests.
> >
> > In all FAIL cases, similar messages appear in xm dmesg.
> >
> > --Keith
> >
> >
> >
> > On Thu, Apr 7, 2011 at 11:23 AM, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com> wrote:
> >> On Thu, Apr 07, 2011 at 11:02:55AM -0500, Keith Swett wrote:
> >>> Local disk, partition, and LVM all work correctly.
> >>
> >> And you mentioned that at the end and I completly missed it. Ooops.
> >> Sorry about that.
> >>
> >> You mentioned Xen 4.0.1... so when did this work? What kernel/hypervisor
> >> did not exhibit this problem?
> >>
> >>
> >>>
> >>> --Keith
> >>>
> >>> On Thu, Apr 7, 2011 at 11:01 AM, Konrad Rzeszutek Wilk
> >>> <konrad.wilk@oracle.com> wrote:
> >>> > On Thu, Apr 07, 2011 at 10:52:37AM -0500, Keith Swett wrote:
> >>> >> A bit more info to add, I just tested with AOE (ata over ethernet)
> >>> >> instead of iSCSI and I see the same issue.
> >>> >
> >>> > Sure, but does it occur if you devices that are not using the network.
> >>> > Like a local disk?
> >>> >
> >>> >>
> >>> >> --Keith
> >>> >>
> >>> >> On Thu, Apr 7, 2011 at 8:54 AM, Konrad Rzeszutek Wilk
> >>> >> <konrad.wilk@oracle.com> wrote:
> >>> >> > On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
> >>> >> >> Hi,
> >>> >> >>
> >>> >> >> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
> >>> >> >> I am importing a iSCSI lun into dom0 from a storage array (using
> >>> >> >> open-iscsi), this is used as the disk [phy:] for my domU.
> >>> >> >
> >>> >> >> If I use HVM, the install works correctly. As soon as I switch to pv
> >>> >> >> for the disk drivers (after installing a xen kernel in the domU) I get
> >>> >> >> a slew of messages in xm dmesg:
> >>> >> > This problem does not occur if the phy is pointing to a local disk?
> >>> >> >>
> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
> >>> >> >> (XEN) printk: 1851 messages suppressed.
> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
> >>> >> >> (XEN) printk: 1354 messages suppressed.
> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
> >>> >> >> (XEN) printk: 1890 messages suppressed.
> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
> >>> >> >> (XEN) printk: 2153 messages suppressed.
> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
> >>> >> >> (XEN) printk: 1912 messages suppressed.
> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
> >>> >> >> (XEN) printk: 2287 messages suppressed.
> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
> >>> >> >> Disk speed is very slow as well. Running DD in the domU to write a
> >>> >> >> file to disk causes more error messages to appear.  It also badly
> >>> >> >> corrupts the disk in some cases.
> >>> >> >> Testing from a local LVM works without error in all cases.
> >>> >> >>
> >>> >> >> Any suggestions?
> >>> >> >> --Keith
> >>> >> >>
> >>> >> >> _______________________________________________
> >>> >> >> Xen-devel mailing list
> >>> >> >> Xen-devel@lists.xensource.com
> >>> >> >> http://lists.xensource.com/xen-devel
> >>> >> >
> >>> >>
> >>> >> _______________________________________________
> >>> >> Xen-devel mailing list
> >>> >> Xen-devel@lists.xensource.com
> >>> >> http://lists.xensource.com/xen-devel
> >>> >
> >>>
> >>> _______________________________________________
> >>> Xen-devel mailing list
> >>> Xen-devel@lists.xensource.com
> >>> http://lists.xensource.com/xen-devel
> >>
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

Not a problem, would be happy to beta test.  Send the patches my way
whenever they're available.
Would you like me to be building against the debian-src of the
packages or would another tree be more beneficial?

--Keith

On Fri, Apr 8, 2011 at 8:53 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Fri, Apr 08, 2011 at 08:40:10AM -0500, Keith Swett wrote:
>> Another update:
>>
>> Retested using Xen 4.0.1-2 (i386 install)
>> and Dom0 Kernel 2.6.32-5-xen-686
>> with the same kernel for the domU.  The same strange behavior still
>> shows up when using iSCSI backing.
>
> So way back at VirtualIron we had some of these issues (if this
> is the same issue) and we fixed it by holding on the pages that
> went to the network stack and took a bit of time to get back.
>
> I was going to dust them of next week to prepare the blkback for
> upstream submission. Are you up for being a beta-tester? Will
> require some kernel recompilation.
>
>>
>> --Keith
>>
>> On Thu, Apr 7, 2011 at 11:39 AM, Keith Swett <swettk@gmail.com> wrote:
>> > Here's a summary:
>> >
>> > All installs are Debian Squeeze
>> >
>> > These stay the same for all tests:
>> > Host is Xen 4.0.1-2 (AMD64 install)
>> > Dom0 Kernel is 2.6.32-5-xen-amd64
>> >
>> > Test procedure:
>> > Install Debian Distro via PXE boot using HVM.
>> > If successful, install xen version of kernel, reboot in non-HVM mode.
>> >
>> > DomUs:
>> >
>> > with iSCSI:
>> > debian pxe netboot installer i386 in HVM mode WORKS (disk detects as
>> > Qemu hard drive)
>> > FAIL after installing a xen kernel (linux-image-xen-686_2.6.32+29) and
>> > switching to builder=linux
>> >
>> > debian pxe netboot installer AMD64 in HVM mode FAIL (disk detects as
>> > Xen Virtual Hard drive)
>> >
>> > with AOE:
>> > debian pxe netboot installer AMD64 in HVM mode FAIL (disk detects as
>> > Xen Virtual Hard drive)
>> >
>> > with LVM:
>> > debian pxe netboot installer i386 in HVM mode WORKS (disk detects as
>> > Qemu hard drive)
>> > WORKS after installing a xen kernel (linux-image-xen-686_2.6.32+29)
>> > and switching to builder=linux
>> >
>> > debian pxe netboot installer AMD64 in HVM mode WORKS (disk detects as
>> > Xen Virtual hard drive)
>> > WORKS after installing a xen kernel
>> > (linux-image-2.6-xen-amd64_2.6.32+29) and switching to builder=linux
>> >
>> > the tests for physical drives are identical to the LVM tests.
>> >
>> > In all FAIL cases, similar messages appear in xm dmesg.
>> >
>> > --Keith
>> >
>> >
>> >
>> > On Thu, Apr 7, 2011 at 11:23 AM, Konrad Rzeszutek Wilk
>> > <konrad.wilk@oracle.com> wrote:
>> >> On Thu, Apr 07, 2011 at 11:02:55AM -0500, Keith Swett wrote:
>> >>> Local disk, partition, and LVM all work correctly.
>> >>
>> >> And you mentioned that at the end and I completly missed it. Ooops.
>> >> Sorry about that.
>> >>
>> >> You mentioned Xen 4.0.1... so when did this work? What kernel/hypervisor
>> >> did not exhibit this problem?
>> >>
>> >>
>> >>>
>> >>> --Keith
>> >>>
>> >>> On Thu, Apr 7, 2011 at 11:01 AM, Konrad Rzeszutek Wilk
>> >>> <konrad.wilk@oracle.com> wrote:
>> >>> > On Thu, Apr 07, 2011 at 10:52:37AM -0500, Keith Swett wrote:
>> >>> >> A bit more info to add, I just tested with AOE (ata over ethernet)
>> >>> >> instead of iSCSI and I see the same issue.
>> >>> >
>> >>> > Sure, but does it occur if you devices that are not using the network.
>> >>> > Like a local disk?
>> >>> >
>> >>> >>
>> >>> >> --Keith
>> >>> >>
>> >>> >> On Thu, Apr 7, 2011 at 8:54 AM, Konrad Rzeszutek Wilk
>> >>> >> <konrad.wilk@oracle.com> wrote:
>> >>> >> > On Wed, Apr 06, 2011 at 11:30:02AM -0500, Keith Swett wrote:
>> >>> >> >> Hi,
>> >>> >> >>
>> >>> >> >> I'm having a strange issue with Xen 4.0.1 AMD64 (debian squeeze).
>> >>> >> >> I am importing a iSCSI lun into dom0 from a storage array (using
>> >>> >> >> open-iscsi), this is used as the disk [phy:] for my domU.
>> >>> >> >
>> >>> >> >> If I use HVM, the install works correctly. As soon as I switch to pv
>> >>> >> >> for the disk drivers (after installing a xen kernel in the domU) I get
>> >>> >> >> a slew of messages in xm dmesg:
>> >>> >> > This problem does not occur if the phy is pointing to a local disk?
>> >>> >> >>
>> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3149
>> >>> >> >> (XEN) printk: 1851 messages suppressed.
>> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3728
>> >>> >> >> (XEN) printk: 1354 messages suppressed.
>> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3485
>> >>> >> >> (XEN) printk: 1890 messages suppressed.
>> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3e58
>> >>> >> >> (XEN) printk: 2153 messages suppressed.
>> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f3c9c
>> >>> >> >> (XEN) printk: 1912 messages suppressed.
>> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f4287
>> >>> >> >> (XEN) printk: 2287 messages suppressed.
>> >>> >> >> (XEN) grant-table.c:1956:d0 Could not get src frame 3f40dc
>> >>> >> >> Disk speed is very slow as well. Running DD in the domU to write a
>> >>> >> >> file to disk causes more error messages to appear.  It also badly
>> >>> >> >> corrupts the disk in some cases.
>> >>> >> >> Testing from a local LVM works without error in all cases.
>> >>> >> >>
>> >>> >> >> Any suggestions?
>> >>> >> >> --Keith
>> >>> >> >>
>> >>> >> >> _______________________________________________
>> >>> >> >> Xen-devel mailing list
>> >>> >> >> Xen-devel@lists.xensource.com
>> >>> >> >> http://lists.xensource.com/xen-devel
>> >>> >> >
>> >>> >>
>> >>> >> _______________________________________________
>> >>> >> Xen-devel mailing list
>> >>> >> Xen-devel@lists.xensource.com
>> >>> >> http://lists.xensource.com/xen-devel
>> >>> >
>> >>>
>> >>> _______________________________________________
>> >>> Xen-devel mailing list
>> >>> Xen-devel@lists.xensource.com
>> >>> http://lists.xensource.com/xen-devel
>> >>
>> >
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

On Fri, Apr 08, 2011 at 08:59:58AM -0500, Keith Swett wrote:
> Not a problem, would be happy to beta test.  Send the patches my way
> whenever they're available.

Did a backport, but I don't have yet the software iSCSI target
available so couldn't do proper tests. Please test and email kernel log
messages.

> Would you like me to be building against the debian-src of the
> packages or would another tree be more beneficial?

That tree should be OK.

See attached patch.

[-- Attachment #2: 89856e4d890a04ce79c6e38c33b5fb873e7fc6df --]
[-- Type: text/plain, Size: 9290 bytes --]

commit 89856e4d890a04ce79c6e38c33b5fb873e7fc6df
Author: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date:   Mon Apr 11 22:37:52 2011 -0400

    blkback: Patch to fix unmapping pages while iSCSI still has references to them.
    
    Signed-off-by: Gary Grebus <ggrebus@virtualiron.com>
    Signed-off-by: Joshua Nicholas <jnicholas@virtualiron.com>
    [v1: Port from dom0-sources.4.5.16.11.tar.gz]
    [v2: Added a whole bunch of printks]
    Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

diff --git a/drivers/xen/blkback/blkback.c b/drivers/xen/blkback/blkback.c
index 0bef445..13c5125 100644
--- a/drivers/xen/blkback/blkback.c
+++ b/drivers/xen/blkback/blkback.c
@@ -81,7 +81,11 @@ typedef struct {
 	atomic_t       pendcnt;
 	unsigned short operation;
 	int            status;
-	struct list_head free_list;
+	unsigned long delay_expire_time;
+	union {
+		struct list_head free_list;
+		struct list_head delayed_list;
+	};
 } pending_req_t;
 
 static pending_req_t *pending_reqs;
@@ -89,6 +93,10 @@ static struct list_head pending_free;
 static DEFINE_SPINLOCK(pending_free_lock);
 static DECLARE_WAIT_QUEUE_HEAD(pending_free_wq);
 
+static struct list_head delayed_reqs;
+static DEFINE_SPINLOCK(delayed_reqs_lock);
+static struct timer_list delayed_reqs_timer;
+
 #define BLKBACK_INVALID_HANDLE (~0)
 
 static struct page **pending_pages;
@@ -132,6 +140,8 @@ static pending_req_t* alloc_req(void)
 		list_del(&req->free_list);
 	}
 	spin_unlock_irqrestore(&pending_free_lock, flags);
+	if (req != NULL)
+		req->delay_expire_time = 0;
 	return req;
 }
 
@@ -169,29 +179,160 @@ static void plug_queue(blkif_t *blkif, struct block_device *bdev)
 	blkif->plug = q;
 }
 
-static void fast_flush_area(pending_req_t *req)
+static void delay_completion(pending_req_t *req)
+{
+	unsigned long flags;
+	pending_req_t *oldest;
+	unsigned long oldest_time;
+
+	req->delay_expire_time = jiffies + (HZ / 64);
+	spin_lock_irqsave(&delayed_reqs_lock, flags);
+	list_add_tail(&req->delayed_list, &delayed_reqs);
+
+	oldest = list_entry(delayed_reqs.next, pending_req_t, delayed_list);
+	oldest_time = oldest->delay_expire_time;
+	spin_unlock_irqrestore(&delayed_reqs_lock, flags);
+
+	mod_timer(&delayed_reqs_timer, oldest_time);
+	printk(KERN_INFO "%s: domid: %d (irq:%u) id: %llx added on delayed list\n",
+			__func__, (unsigned int)req->blkif->domid,
+			        req->blkif->irq,
+				req->id);
+}
+
+static int fast_flush_area(pending_req_t *req)
 {
 	struct gnttab_unmap_grant_ref unmap[BLKIF_MAX_SEGMENTS_PER_REQUEST];
 	unsigned int i, invcount = 0;
 	grant_handle_t handle;
 	int ret;
 
+	struct page *pending_page;
+	struct page *new_page;
+	int pending_page_count;
+	int pending_page_mapcount;
+
+	unsigned int pgidx;
+	unsigned int busy_pages = 0;
+	int delay_expired;
+
+	/* Already waited for extra refs to clear on one or more pages? */
+	delay_expired = req->delay_expire_time &&
+		time_before(req->delay_expire_time, jiffies);
+
 	for (i = 0; i < req->nr_pages; i++) {
 		handle = pending_handle(req, i);
 		if (handle == BLKBACK_INVALID_HANDLE)
 			continue;
-		blkback_pagemap_clear(pending_page(req, i));
-		gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i),
-				    GNTMAP_host_map, handle);
-		pending_handle(req, i) = BLKBACK_INVALID_HANDLE;
-		invcount++;
+
+		pgidx = vaddr_pagenr(req, i);
+		pending_page = pending_pages[pgidx];
+		pending_page_count = page_count(pending_page);
+		pending_page_mapcount = page_mapcount(pending_page);
+
+		if (pending_page_count <= 1) {
+			if (unlikely(pending_page_mapcount > 0)) {	// Used as a flag by netback lazy copying scheme,
+				reset_page_mapcount(pending_page);		// but blocks gnttab_copy_grant_page() if set
+			}
+			blkback_pagemap_clear(pending_page(req, i));
+			gnttab_set_unmap_op(&unmap[invcount++], vaddr(req, i),
+					    GNTMAP_host_map, handle);
+			pending_handle(req, i) = BLKBACK_INVALID_HANDLE;
+		} else {
+			printk(KERN_INFO "domid: %d (irq:%u) id: %llx count: %u, mapcount: %u, delay %s, busy_pages:%d\n",
+				(unsigned int)req->blkif->domid,
+			        req->blkif->irq,
+				req->id,
+				pending_page_count,
+				pending_page_mapcount,
+				delay_expired ? "expire" : "still ticking",
+				busy_pages);
+			if (!delay_expired) {
+				busy_pages++;
+			} else {
+				if (unlikely(pending_page_mapcount > 0)) {	// Used as a flag by netback lazy copying scheme,
+					reset_page_mapcount(pending_page);		// but blocks gnttab_copy_grant_page() if set
+					busy_pages++;
+				} else {
+					printk(KERN_INFO "domid: %d (irq:%u) id: %llx leaking page %d\n",
+			       			(unsigned int)req->blkif->domid,
+			       			req->blkif->irq, req->id,
+						pgidx);			
+					new_page = pending_page;
+					ret = gnttab_copy_grant_page(handle, &new_page);	// new dom0 page is returned
+					if ( ! ret ) {
+						pending_pages[pgidx] = new_page;
+						pending_handle(req, i) = BLKBACK_INVALID_HANDLE;
+					} else {
+						busy_pages++;
+					}
+				}
+			}
+		}
 	}
 
-	ret = HYPERVISOR_grant_table_op(
-		GNTTABOP_unmap_grant_ref, unmap, invcount);
-	BUG_ON(ret);
+	if (invcount > 0) {
+		ret = HYPERVISOR_grant_table_op(
+			GNTTABOP_unmap_grant_ref, unmap, invcount);
+		BUG_ON(ret);
+	}
+	return busy_pages;
 }
 
+
+/*
+ * Completion of a request is delayed if one or more of the mapped guest
+ * pages still has a reference.  This can happen if the block I/O gets
+ * turned into a network I/O for example with iSCSI.
+ */
+static void complete_delayed_reqs(unsigned long unused)
+{
+	pending_req_t *req;
+	pending_req_t *tmp;
+	unsigned long oldest_time = 0;
+	unsigned long flags;
+	unsigned long count = 0;
+	LIST_HEAD(work_list);
+
+	if (list_empty(&delayed_reqs))
+		return;
+
+	/* Grab the whole list */
+	spin_lock_irqsave(&delayed_reqs_lock, flags);
+	list_for_each_entry_safe(req, tmp, &delayed_reqs, delayed_list) {
+		count++;
+		list_move_tail(&req->delayed_list, &work_list);
+	}
+	spin_unlock_irqrestore(&delayed_reqs_lock, flags);
+	printk(KERN_INFO "%s: processing %ld requests.\n", __func__, count);
+	list_for_each_entry_safe(req, tmp, &work_list, delayed_list) {
+		printk(KERN_INFO "%s: domid: %d (irq:%u) id: %llx\n", __func__,
+			(unsigned int)req->blkif->domid,
+			req->blkif->irq, req->id);
+		if (unlikely(fast_flush_area(req))) {
+			spin_lock_irqsave(&delayed_reqs_lock, flags);
+			list_move_tail(&req->delayed_list, &delayed_reqs);
+			spin_unlock_irqrestore(&delayed_reqs_lock, flags);
+		} else {
+			list_del(&req->delayed_list);
+			make_response(req->blkif, req->id,
+				      req->operation, req->status);
+
+			blkif_put(req->blkif);
+			free_req(req);
+		}
+	}
+
+	spin_lock_irqsave(&delayed_reqs_lock, flags);
+	if (! list_empty(&delayed_reqs)) {
+		tmp = list_entry(delayed_reqs.next, pending_req_t,
+				 delayed_list);
+		oldest_time = tmp->delay_expire_time;
+	}
+	spin_unlock_irqrestore(&delayed_reqs_lock, flags);
+	if (oldest_time)
+		mod_timer(&delayed_reqs_timer, oldest_time);
+}
 /******************************************************************
  * SCHEDULER FUNCTIONS
  */
@@ -258,6 +399,10 @@ int blkif_schedule(void *arg)
 
 static void __end_block_io_op(pending_req_t *pending_req, int error)
 {
+
+
+	complete_delayed_reqs(0); /* Previously delayed reqs are usually done.*/
+
 	/* An error fails the entire request. */
 	if ((pending_req->operation == BLKIF_OP_WRITE_BARRIER) &&
 	    (error == -EOPNOTSUPP)) {
@@ -271,9 +416,16 @@ static void __end_block_io_op(pending_req_t *pending_req, int error)
 	}
 
 	if (atomic_dec_and_test(&pending_req->pendcnt)) {
-		fast_flush_area(pending_req);
+				pending_req->id, duration);
+		if (unlikely(fast_flush_area(pending_req))) {
+			/* Granted page(s) are still being referenced */
+			delay_completion(pending_req);
+			return;
+		}
+
 		make_response(pending_req->blkif, pending_req->id,
 			      pending_req->operation, pending_req->status);
+
 		blkif_put(pending_req->blkif);
 		free_req(pending_req);
 	}
@@ -316,6 +468,7 @@ static int do_block_io_op(blkif_t *blkif)
 	RING_IDX rc, rp;
 	int more_to_do = 0;
 
+ 
 	rc = blk_rings->common.req_cons;
 	rp = blk_rings->common.sring->req_prod;
 	rmb(); /* Ensure we see queued requests up to 'rp'. */
@@ -337,6 +490,7 @@ static int do_block_io_op(blkif_t *blkif)
 			break;
 		}
 
+
 		switch (blkif->blk_protocol) {
 		case BLKIF_PROTOCOL_NATIVE:
 			memcpy(&req, RING_GET_REQUEST(&blk_rings->native, rc), sizeof(req));
@@ -542,7 +696,13 @@ static void dispatch_rw_block_io(blkif_t *blkif,
 	return;
 
  fail_flush:
-	fast_flush_area(pending_req);
+	if (unlikely(fast_flush_area(pending_req))) {
+		/* Granted page(s) are being referenced by a previous I/O */
+		pending_req->status = BLKIF_RSP_ERROR;
+		delay_completion(pending_req);
+		msleep(1); /* back off a bit */
+		return;
+	}	
  fail_response:
 	make_response(blkif, req->id, req->operation, BLKIF_RSP_ERROR);
 	free_req(pending_req);
@@ -652,6 +812,12 @@ static int __init blkif_init(void)
 	memset(pending_reqs, 0, sizeof(pending_reqs));
 	INIT_LIST_HEAD(&pending_free);
 
+	INIT_LIST_HEAD(&delayed_reqs);
+	spin_lock_init(&delayed_reqs_lock);
+	init_timer(&delayed_reqs_timer);
+	delayed_reqs_timer.data = 0;
+	delayed_reqs_timer.function = complete_delayed_reqs;
+
 	for (i = 0; i < blkif_reqs; i++)
 		list_add_tail(&pending_reqs[i].free_list, &pending_free);
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

Konrad,

That seems to have fixed it, i'm not seeing the error messages anymore
and performance seems correct thus far.  I'll continue to test a
couple more scenarios today and let you know if I see anything odd.
How should we go about getting this over to the debian maintainers?

--Keith

On Tue, Apr 12, 2011 at 5:20 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Fri, Apr 08, 2011 at 08:59:58AM -0500, Keith Swett wrote:
>> Not a problem, would be happy to beta test.  Send the patches my way
>> whenever they're available.
>
> Did a backport, but I don't have yet the software iSCSI target
> available so couldn't do proper tests. Please test and email kernel log
> messages.
>
>> Would you like me to be building against the debian-src of the
>> packages or would another tree be more beneficial?
>
> That tree should be OK.
>
> See attached patch.
>

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

On Tue, Apr 12, 2011 at 11:20:07AM -0500, Keith Swett wrote:
> Konrad,
> 
> That seems to have fixed it, i'm not seeing the error messages anymore
> and performance seems correct thus far.  I'll continue to test a
> couple more scenarios today and let you know if I see anything odd.

OK. It might take a bit of time. You should see some kernel logs when
it detects the iSCSI retransmission errors. Why don't we let this
run for a week or so to make sure it does not cause regressions.

> How should we go about getting this over to the debian maintainers?

Lets make sure it does not break anything.
> 
> --Keith
> 
> On Tue, Apr 12, 2011 at 5:20 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Fri, Apr 08, 2011 at 08:59:58AM -0500, Keith Swett wrote:
> >> Not a problem, would be happy to beta test.  Send the patches my way
> >> whenever they're available.
> >
> > Did a backport, but I don't have yet the software iSCSI target
> > available so couldn't do proper tests. Please test and email kernel log
> > messages.
> >
> >> Would you like me to be building against the debian-src of the
> >> packages or would another tree be more beneficial?
> >
> > That tree should be OK.
> >
> > See attached patch.
> >

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

Konrad,

I'm not seeing the kernel messages, so i'm wondering if something else
changed, and if I applied the patch correctly.
Trying to repeat the process from this morning with a clean tree to confirm.


--Keith


On Tue, Apr 12, 2011 at 3:41 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Tue, Apr 12, 2011 at 11:20:07AM -0500, Keith Swett wrote:
>> Konrad,
>>
>> That seems to have fixed it, i'm not seeing the error messages anymore
>> and performance seems correct thus far.  I'll continue to test a
>> couple more scenarios today and let you know if I see anything odd.
>
> OK. It might take a bit of time. You should see some kernel logs when
> it detects the iSCSI retransmission errors. Why don't we let this
> run for a week or so to make sure it does not cause regressions.
>
>> How should we go about getting this over to the debian maintainers?
>
> Lets make sure it does not break anything.
>>
>> --Keith
>>
>> On Tue, Apr 12, 2011 at 5:20 AM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Fri, Apr 08, 2011 at 08:59:58AM -0500, Keith Swett wrote:
>> >> Not a problem, would be happy to beta test.  Send the patches my way
>> >> whenever they're available.
>> >
>> > Did a backport, but I don't have yet the software iSCSI target
>> > available so couldn't do proper tests. Please test and email kernel log
>> > messages.
>> >
>> >> Would you like me to be building against the debian-src of the
>> >> packages or would another tree be more beneficial?
>> >
>> > That tree should be OK.
>> >
>> > See attached patch.
>> >
>

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

On Tue, Apr 12, 2011 at 03:59:48PM -0500, Keith Swett wrote:
> Konrad,
> 
> I'm not seeing the kernel messages, so i'm wondering if something else
> changed, and if I applied the patch correctly.

Make sure you have 'loglevel=8 debug' in your Linux command. Also you might
need to look in /var/log/kern during runtime (or maybe some other place?)

> Trying to repeat the process from this morning with a clean tree to confirm.

OK.

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

Konrad,
I found an error in my process.
On line 225 of the patch, there's a single parenthesis.  What should
the lines in this area (418-420 of blkback.c) look like after the
patch?

--Keith

On Tue, Apr 12, 2011 at 4:08 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Tue, Apr 12, 2011 at 03:59:48PM -0500, Keith Swett wrote:
>> Konrad,
>>
>> I'm not seeing the kernel messages, so i'm wondering if something else
>> changed, and if I applied the patch correctly.
>
> Make sure you have 'loglevel=8 debug' in your Linux command. Also you might
> need to look in /var/log/kern during runtime (or maybe some other place?)
>
>> Trying to repeat the process from this morning with a clean tree to confirm.
>
> OK.
>

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

On Tue, Apr 12, 2011 at 04:23:13PM -0500, Keith Swett wrote:
> Konrad,

Please don't top-post.
> I found an error in my process.
> On line 225 of the patch, there's a single parenthesis.  What should
> the lines in this area (418-420 of blkback.c) look like after the

Here is the git tree with the patch:

http://git.kernel.org/?p=linux/kernel/git/konrad/xen.git;a=shortlog;h=refs/heads/devel/blkback-vi
And the file
http://git.kernel.org/?p=linux/kernel/git/konrad/xen.git;a=blob;f=drivers/xen/blkback/blkback.c;h=13c51257f2d4f87330f13820a372abcff63864b4;hb=89856e4d890a04ce79c6e38c33b5fb873e7fc6df


And I think you are referring to this:
 417 
 418         if (atomic_dec_and_test(&pending_req->pendcnt)) {
 419                                 pending_req->id, duration); <<---
 420                 if (unlikely(fast_flush_area(pending_req))) {
 421                         /* Granted page(s) are still being referenc

Take out line 419.

Re: Trying to track down odd behavior, errors in xen dmesg

[Keith Swett]

Konrad,

> Please don't top-post.
Sorry.

I did comment out the line (I didn't see the duration variable
declared anywhere)
and rebuilt and re-tested.  It appears to have the same error.  I
found the problem with this morning's testing.  The error occurs when
I place the VM running the iSCSI target on the same physical machine
that is running the initiator (this may seem backwards but it's for a
DRBD/Pacemaker fail-over)  Is this just an unsupported configuration?

--Keith

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

On Tue, Apr 12, 2011 at 06:02:43PM -0500, Keith Swett wrote:
> Konrad,
> 
> > Please don't top-post.
> Sorry.
> 
> I did comment out the line (I didn't see the duration variable
> declared anywhere)
> and rebuilt and re-tested.  It appears to have the same error.  I

OK, were there are any other errors?
> found the problem with this morning's testing.  The error occurs when
> I place the VM running the iSCSI target on the same physical machine
> that is running the initiator (this may seem backwards but it's for a

Oh wow. iSCSI target and iSCSI initiator on the same box and then DRBD on top?
Whoa.

> DRBD/Pacemaker fail-over)  Is this just an unsupported configuration?

IT is an odd configuration.
> 
> --Keith
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Trying to track down odd behavior, errors in xen dmesg

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]

On Tue, Apr 12, 2011 at 04:23:13PM -0500, Keith Swett wrote:
> Konrad,
> I found an error in my process.
> On line 225 of the patch, there's a single parenthesis.  What should
> the lines in this area (418-420 of blkback.c) look like after the
> patch?

I updated the patch. Try the attached one.
> 
> --Keith
> 
> On Tue, Apr 12, 2011 at 4:08 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Tue, Apr 12, 2011 at 03:59:48PM -0500, Keith Swett wrote:
> >> Konrad,
> >>
> >> I'm not seeing the kernel messages, so i'm wondering if something else
> >> changed, and if I applied the patch correctly.
> >
> > Make sure you have 'loglevel=8 debug' in your Linux command. Also you might
> > need to look in /var/log/kern during runtime (or maybe some other place?)
> >
> >> Trying to repeat the process from this morning with a clean tree to confirm.
> >
> > OK.
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 9285 bytes --]

commit ff1902ee8462cf7d763a8fd42255a72f8e31bebe
Author: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Date:   Mon Apr 11 22:37:52 2011 -0400

    blkback: Patch to fix unmapping pages while iSCSI still has references to them.
    
    Signed-off-by: Gary Grebus <ggrebus@virtualiron.com>
    Signed-off-by: Joshua Nicholas <jnicholas@virtualiron.com>
    [v1: Port from dom0-sources.4.5.16.11.tar.gz]
    [v2: Added a whole bunch of printks]
    [v3: Fixed merge error]
    Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

diff --git a/drivers/xen/blkback/blkback.c b/drivers/xen/blkback/blkback.c
index 0bef445..ab8d12d 100644
--- a/drivers/xen/blkback/blkback.c
+++ b/drivers/xen/blkback/blkback.c
@@ -81,7 +81,11 @@ typedef struct {
 	atomic_t       pendcnt;
 	unsigned short operation;
 	int            status;
-	struct list_head free_list;
+	unsigned long delay_expire_time;
+	union {
+		struct list_head free_list;
+		struct list_head delayed_list;
+	};
 } pending_req_t;
 
 static pending_req_t *pending_reqs;
@@ -89,6 +93,10 @@ static struct list_head pending_free;
 static DEFINE_SPINLOCK(pending_free_lock);
 static DECLARE_WAIT_QUEUE_HEAD(pending_free_wq);
 
+static struct list_head delayed_reqs;
+static DEFINE_SPINLOCK(delayed_reqs_lock);
+static struct timer_list delayed_reqs_timer;
+
 #define BLKBACK_INVALID_HANDLE (~0)
 
 static struct page **pending_pages;
@@ -132,6 +140,8 @@ static pending_req_t* alloc_req(void)
 		list_del(&req->free_list);
 	}
 	spin_unlock_irqrestore(&pending_free_lock, flags);
+	if (req != NULL)
+		req->delay_expire_time = 0;
 	return req;
 }
 
@@ -169,29 +179,160 @@ static void plug_queue(blkif_t *blkif, struct block_device *bdev)
 	blkif->plug = q;
 }
 
-static void fast_flush_area(pending_req_t *req)
+static void delay_completion(pending_req_t *req)
+{
+	unsigned long flags;
+	pending_req_t *oldest;
+	unsigned long oldest_time;
+
+	req->delay_expire_time = jiffies + (HZ / 64);
+	spin_lock_irqsave(&delayed_reqs_lock, flags);
+	list_add_tail(&req->delayed_list, &delayed_reqs);
+
+	oldest = list_entry(delayed_reqs.next, pending_req_t, delayed_list);
+	oldest_time = oldest->delay_expire_time;
+	spin_unlock_irqrestore(&delayed_reqs_lock, flags);
+
+	mod_timer(&delayed_reqs_timer, oldest_time);
+	printk(KERN_INFO "%s: domid: %d (irq:%u) id: %llx added on delayed list\n",
+			__func__, (unsigned int)req->blkif->domid,
+			        req->blkif->irq,
+				req->id);
+}
+
+static int fast_flush_area(pending_req_t *req)
 {
 	struct gnttab_unmap_grant_ref unmap[BLKIF_MAX_SEGMENTS_PER_REQUEST];
 	unsigned int i, invcount = 0;
 	grant_handle_t handle;
 	int ret;
 
+	struct page *pending_page;
+	struct page *new_page;
+	int pending_page_count;
+	int pending_page_mapcount;
+
+	unsigned int pgidx;
+	unsigned int busy_pages = 0;
+	int delay_expired;
+
+	/* Already waited for extra refs to clear on one or more pages? */
+	delay_expired = req->delay_expire_time &&
+		time_before(req->delay_expire_time, jiffies);
+
 	for (i = 0; i < req->nr_pages; i++) {
 		handle = pending_handle(req, i);
 		if (handle == BLKBACK_INVALID_HANDLE)
 			continue;
-		blkback_pagemap_clear(pending_page(req, i));
-		gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i),
-				    GNTMAP_host_map, handle);
-		pending_handle(req, i) = BLKBACK_INVALID_HANDLE;
-		invcount++;
+
+		pgidx = vaddr_pagenr(req, i);
+		pending_page = pending_pages[pgidx];
+		pending_page_count = page_count(pending_page);
+		pending_page_mapcount = page_mapcount(pending_page);
+
+		if (pending_page_count <= 1) {
+			if (unlikely(pending_page_mapcount > 0)) {	// Used as a flag by netback lazy copying scheme,
+				reset_page_mapcount(pending_page);		// but blocks gnttab_copy_grant_page() if set
+			}
+			blkback_pagemap_clear(pending_page(req, i));
+			gnttab_set_unmap_op(&unmap[invcount++], vaddr(req, i),
+					    GNTMAP_host_map, handle);
+			pending_handle(req, i) = BLKBACK_INVALID_HANDLE;
+		} else {
+			printk(KERN_INFO "domid: %d (irq:%u) id: %llx count: %u, mapcount: %u, delay %s, busy_pages:%d\n",
+				(unsigned int)req->blkif->domid,
+			        req->blkif->irq,
+				req->id,
+				pending_page_count,
+				pending_page_mapcount,
+				delay_expired ? "expire" : "still ticking",
+				busy_pages);
+			if (!delay_expired) {
+				busy_pages++;
+			} else {
+				if (unlikely(pending_page_mapcount > 0)) {	// Used as a flag by netback lazy copying scheme,
+					reset_page_mapcount(pending_page);		// but blocks gnttab_copy_grant_page() if set
+					busy_pages++;
+				} else {
+					printk(KERN_INFO "domid: %d (irq:%u) id: %llx leaking page %d\n",
+			       			(unsigned int)req->blkif->domid,
+			       			req->blkif->irq, req->id,
+						pgidx);			
+					new_page = pending_page;
+					ret = gnttab_copy_grant_page(handle, &new_page);	// new dom0 page is returned
+					if ( ! ret ) {
+						pending_pages[pgidx] = new_page;
+						pending_handle(req, i) = BLKBACK_INVALID_HANDLE;
+					} else {
+						busy_pages++;
+					}
+				}
+			}
+		}
 	}
 
-	ret = HYPERVISOR_grant_table_op(
-		GNTTABOP_unmap_grant_ref, unmap, invcount);
-	BUG_ON(ret);
+	if (invcount > 0) {
+		ret = HYPERVISOR_grant_table_op(
+			GNTTABOP_unmap_grant_ref, unmap, invcount);
+		BUG_ON(ret);
+	}
+	return busy_pages;
 }
 
+
+/*
+ * Completion of a request is delayed if one or more of the mapped guest
+ * pages still has a reference.  This can happen if the block I/O gets
+ * turned into a network I/O for example with iSCSI.
+ */
+static void complete_delayed_reqs(unsigned long unused)
+{
+	pending_req_t *req;
+	pending_req_t *tmp;
+	unsigned long oldest_time = 0;
+	unsigned long flags;
+	unsigned long count = 0;
+	LIST_HEAD(work_list);
+
+	if (list_empty(&delayed_reqs))
+		return;
+
+	/* Grab the whole list */
+	spin_lock_irqsave(&delayed_reqs_lock, flags);
+	list_for_each_entry_safe(req, tmp, &delayed_reqs, delayed_list) {
+		count++;
+		list_move_tail(&req->delayed_list, &work_list);
+	}
+	spin_unlock_irqrestore(&delayed_reqs_lock, flags);
+	printk(KERN_INFO "%s: processing %ld requests.\n", __func__, count);
+	list_for_each_entry_safe(req, tmp, &work_list, delayed_list) {
+		printk(KERN_INFO "%s: domid: %d (irq:%u) id: %llx\n", __func__,
+			(unsigned int)req->blkif->domid,
+			req->blkif->irq, req->id);
+		if (unlikely(fast_flush_area(req))) {
+			spin_lock_irqsave(&delayed_reqs_lock, flags);
+			list_move_tail(&req->delayed_list, &delayed_reqs);
+			spin_unlock_irqrestore(&delayed_reqs_lock, flags);
+		} else {
+			list_del(&req->delayed_list);
+			make_response(req->blkif, req->id,
+				      req->operation, req->status);
+
+			blkif_put(req->blkif);
+			free_req(req);
+		}
+	}
+
+	spin_lock_irqsave(&delayed_reqs_lock, flags);
+	if (! list_empty(&delayed_reqs)) {
+		tmp = list_entry(delayed_reqs.next, pending_req_t,
+				 delayed_list);
+		oldest_time = tmp->delay_expire_time;
+	}
+	spin_unlock_irqrestore(&delayed_reqs_lock, flags);
+	if (oldest_time)
+		mod_timer(&delayed_reqs_timer, oldest_time);
+}
 /******************************************************************
  * SCHEDULER FUNCTIONS
  */
@@ -258,6 +399,10 @@ int blkif_schedule(void *arg)
 
 static void __end_block_io_op(pending_req_t *pending_req, int error)
 {
+
+
+	complete_delayed_reqs(0); /* Previously delayed reqs are usually done.*/
+
 	/* An error fails the entire request. */
 	if ((pending_req->operation == BLKIF_OP_WRITE_BARRIER) &&
 	    (error == -EOPNOTSUPP)) {
@@ -271,9 +416,15 @@ static void __end_block_io_op(pending_req_t *pending_req, int error)
 	}
 
 	if (atomic_dec_and_test(&pending_req->pendcnt)) {
-		fast_flush_area(pending_req);
+		if (unlikely(fast_flush_area(pending_req))) {
+			/* Granted page(s) are still being referenced */
+			delay_completion(pending_req);
+			return;
+		}
+
 		make_response(pending_req->blkif, pending_req->id,
 			      pending_req->operation, pending_req->status);
+
 		blkif_put(pending_req->blkif);
 		free_req(pending_req);
 	}
@@ -316,6 +467,7 @@ static int do_block_io_op(blkif_t *blkif)
 	RING_IDX rc, rp;
 	int more_to_do = 0;
 
+ 
 	rc = blk_rings->common.req_cons;
 	rp = blk_rings->common.sring->req_prod;
 	rmb(); /* Ensure we see queued requests up to 'rp'. */
@@ -337,6 +489,7 @@ static int do_block_io_op(blkif_t *blkif)
 			break;
 		}
 
+
 		switch (blkif->blk_protocol) {
 		case BLKIF_PROTOCOL_NATIVE:
 			memcpy(&req, RING_GET_REQUEST(&blk_rings->native, rc), sizeof(req));
@@ -542,7 +695,13 @@ static void dispatch_rw_block_io(blkif_t *blkif,
 	return;
 
  fail_flush:
-	fast_flush_area(pending_req);
+	if (unlikely(fast_flush_area(pending_req))) {
+		/* Granted page(s) are being referenced by a previous I/O */
+		pending_req->status = BLKIF_RSP_ERROR;
+		delay_completion(pending_req);
+		msleep(1); /* back off a bit */
+		return;
+	}	
  fail_response:
 	make_response(blkif, req->id, req->operation, BLKIF_RSP_ERROR);
 	free_req(pending_req);
@@ -652,6 +811,12 @@ static int __init blkif_init(void)
 	memset(pending_reqs, 0, sizeof(pending_reqs));
 	INIT_LIST_HEAD(&pending_free);
 
+	INIT_LIST_HEAD(&delayed_reqs);
+	spin_lock_init(&delayed_reqs_lock);
+	init_timer(&delayed_reqs_timer);
+	delayed_reqs_timer.data = 0;
+	delayed_reqs_timer.function = complete_delayed_reqs;
+
 	for (i = 0; i < blkif_reqs; i++)
 		list_add_tail(&pending_reqs[i].free_list, &pending_free);
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel