From mboxrd@z Thu Jan 1 00:00:00 1970 From: greno@verizon.net Subject: Xen pv_ops dom0 2.6.32.13 issues Date: Wed, 09 Jun 2010 16:02:11 -0500 (CDT) Message-ID: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0930152500==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: greno@verizon.net Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org --===============0930152500== Content-type: text/html; charset=UTF-8 Content-transfer-encoding: quoted-printable
Ok, I've been running this 2.6.32.13 pv_ops dom0 kernel for several wee= ks and it has twice killed my domU's.  I get numerous CPU soft lockup = bug errors and at times it will freeze which means a power cycle boot. = ; This has resulted in things like:
EXT-fs error (device dm-0): ext4_loo= kup: deleted inode reference
EXT-fs error (device dm-0): ext4_lookup: de= leted inode reference
in the domU boots which has killed two of them.&nb= sp; I had to rebuild the images from backups.  The domU's are running = 2.6.32-16-server domU kernels (ubuntu).


-Gerry


May 19= , 2010 09:01:00 PM, greno@verizon.net wrote:
On 05/19/2010 05:24 PM, Jeremy Fitzha= rdinge wrote:
> On 05/19/2010 02:09 PM, Gerry Reno wrote:
> =
>> I am using a pv_ops dom0 kernel 2.6.32.12 with xen 4.0.0-rc8. = My
>> domU's use 2.6.31-14-server ubuntu.
>>
>>= When I try to ping another computer on the network from the domU I
>= > still received this error:
>> Attempting to checksum a non-TC= P/UDP packet, dropping a protocol 1 packet
>>
>> I though= t this error was fixed somewhere around 2.6.32.10 but
>> apparentl= y it is still in 2.6.32.12.
>>
>> How do I get around thi= s problem?
>>
> I applied a patch to fix up a checksum= bug in netback, but I realized I
> hadn't applied it to stable-2.6.3= 2. Please try again (it will be
> 2.6.32.13) and tell me how it goes= .
>
> Thanks,
> J
>
>
Jeremy, th= at fixed it. Thanks.

-Gerry


____________________________= ___________________
Xen-devel mailing list
Xen-devel@lists.xensource.= com
http://lists.xensource.com/xen-devel
--===============0930152500== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0930152500==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: Xen pv_ops dom0 2.6.32.13 issues Date: Wed, 09 Jun 2010 15:37:38 -0700 Message-ID: <4C1017B2.90800@goop.org> References: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: greno@verizon.net Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org On 06/09/2010 02:02 PM, greno@verizon.net wrote: > > Ok, I've been running this 2.6.32.13 pv_ops dom0 kernel for several > weeks and it has twice killed my domU's. I get numerous CPU soft > lockup bug errors and at times it will freeze which means a power > cycle boot. The lockups are in dom0 or domU? Do the backtraces indicate a common subsystem, or are they all over the place? > This has resulted in things like: > EXT-fs error (device dm-0): ext4_lookup: deleted inode reference > EXT-fs error (device dm-0): ext4_lookup: deleted inode reference > in the domU boots which has killed two of them. What's your storage path from guest device to media? Are they using barriers? J From mboxrd@z Thu Jan 1 00:00:00 1970 From: greno@verizon.net Subject: Re: Re: Xen pv_ops dom0 2.6.32.13 issues Date: Wed, 09 Jun 2010 18:27:33 -0500 (CDT) Message-ID: <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_4840452_29235966.1276126053808" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: jeremy@goop.org Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org ------=_Part_4840452_29235966.1276126053808 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit blkbackd



Jun 9, 2010 07:13:23 PM, jeremy@goop.org wrote:
On 06/09/2010 04:05 PM, greno@verizon.net wrote:
> Jeremy,
> The soft lockups seemed to be occurring in different systems. And I
> could never make sense out of what was triggering them. I have not
> mounted any file systems with "nobarriers" in guests. The guests are
> all a single /dev/xvda. The underlying physical hardware is LVM over
> RAID-1 arrays. I'm attaching dmesg, kern.log, and messages in case
> these might be useful.

Using what storage backend? blkback? blktap2?

J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.co m
http://lists.xensource.com/xen-devel
------=_Part_4840452_29235966.1276126053808 Content-Type: application/octet-stream; name=blkbackd.log Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=blkbackd.log xenstore_scan: /local/domain/0/backend/blkbackd quit on signal: 15 ------=_Part_4840452_29235966.1276126053808 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ------=_Part_4840452_29235966.1276126053808-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: Xen pv_ops dom0 2.6.32.13 issues Date: Wed, 09 Jun 2010 16:37:36 -0700 Message-ID: <4C1025C0.2070808@goop.org> References: <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: greno@verizon.net Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org On 06/09/2010 04:27 PM, greno@verizon.net wrote: > blkbackd Using phy: in your config file? That really isn't recommended because it has poor integrity; the writes are buffered in dom0 so writes can be reordered or lost on crash, and the guest filesystem can't maintain any of its own integrity guarantees. tap:aio: is more resilient, since the writes go directly to the device without buffering. That doesn't directly relate to your lockup issues, but it should prevent filesystem corruption when they happen. J > > > > Jun 9, 2010 07:13:23 PM, jeremy@goop.org wrote: > > On 06/09/2010 04:05 PM, greno@verizon.net wrote: > > Jeremy, > > The soft lockups seemed to be occurring in different systems. And I > > could never make sense out of what was triggering them. I have not > > mounted any file systems with "nobarriers" in guests. The guests are > > all a single /dev/xvda. The underlying physical hardware is LVM over > > RAID-1 arrays. I'm attaching dmesg, kern.log, and messages in case > > these might be useful. > > Using what storage backend? blkback? blktap2? > > J > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Neobiker Subject: Which disk backend to use in domU? Date: Fri, 11 Jun 2010 10:06:38 -0700 (PDT) Message-ID: <28857720.post@talk.nabble.com> References: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> <4C1025C0.2070808@goop.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4C1025C0.2070808@goop.org> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Hello Jeremy, Jeremy Fitzhardinge wrote: > Using phy: in your config file? That really isn't recommended because it > has poor integrity; the writes are buffered in dom0 so writes can be > reordered or lost on crash, and the guest filesystem can't maintain any > of its own integrity guarantees. > > tap:aio: is more resilient, since the writes go directly to the device > without buffering. Do you mean that using tap:aio with a disk.image is prefered against using phy: with LVM-device? Best Regards Jens Friedrich aka Neobiker (www.neobiker.de) -- View this message in context: http://old.nabble.com/Xen-pv_ops-dom0-2.6.32.13-issues-tp28835895p28857720.html Sent from the Xen - Dev mailing list archive at Nabble.com. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Valtteri Kiviniemi Subject: Re: Which disk backend to use in domU? Date: Fri, 11 Jun 2010 20:42:28 +0300 Message-ID: <4C127584.6070801@dataproof.fi> References: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> <4C1025C0.2070808@goop.org> <28857720.post@talk.nabble.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <28857720.post@talk.nabble.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Hi, I am also using phy: with LVM-partitions, and I also would like to know if there is a better or more preferred way. - Valtteri Kiviniemi Neobiker kirjoitti: > Hello Jeremy, > > > Jeremy Fitzhardinge wrote: >> Using phy: in your config file? That really isn't recommended because it >> has poor integrity; the writes are buffered in dom0 so writes can be >> reordered or lost on crash, and the guest filesystem can't maintain any >> of its own integrity guarantees. >> >> tap:aio: is more resilient, since the writes go directly to the device >> without buffering. > > Do you mean that using tap:aio with a disk.image is prefered against using > phy: with LVM-device? > > Best Regards > Jens Friedrich aka Neobiker (www.neobiker.de) From mboxrd@z Thu Jan 1 00:00:00 1970 From: Valtteri Kiviniemi Subject: Re: Which disk backend to use in domU? Date: Fri, 11 Jun 2010 20:53:30 +0300 Message-ID: <4C12781A.7060603@dataproof.fi> References: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> <4C1025C0.2070808@goop.org> <28857720.post@talk.nabble.com> <4C127584.6070801@dataproof.fi> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4C127584.6070801@dataproof.fi> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Hi, Ah, misunderstanding sorry, you were talking about disk images :) Valtteri Kiviniemi kirjoitti: > Hi, > > I am also using phy: with LVM-partitions, and I also would like to know > if there is a better or more preferred way. > > - Valtteri Kiviniemi > > Neobiker kirjoitti: >> Hello Jeremy, >> >> >> Jeremy Fitzhardinge wrote: >>> Using phy: in your config file? That really isn't recommended >>> because it >>> has poor integrity; the writes are buffered in dom0 so writes can be >>> reordered or lost on crash, and the guest filesystem can't maintain any >>> of its own integrity guarantees. >>> >>> tap:aio: is more resilient, since the writes go directly to the device >>> without buffering. >> >> Do you mean that using tap:aio with a disk.image is prefered against >> using >> phy: with LVM-device? >> >> Best Regards >> Jens Friedrich aka Neobiker (www.neobiker.de) > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: Neobiker Subject: Re: Which disk backend to use in domU? Date: Fri, 11 Jun 2010 11:11:42 -0700 (PDT) Message-ID: <28858517.post@talk.nabble.com> References: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> <4C1025C0.2070808@goop.org> <28857720.post@talk.nabble.com> <4C127584.6070801@dataproof.fi> <4C12781A.7060603@dataproof.fi> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4C12781A.7060603@dataproof.fi> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Hi Valtteri Kiviniemi-2 wrote: > > Hi, Ah, misunderstanding sorry, you were talking about disk images :) > I'm talking about this config: disk = [ 'phy:/dev/vm/vm01,xvda1,w', 'phy:/dev/vm/vm01-swap,xvda2,w', 'phy:/dev/daten/devel_debian_amd64,xvda3,w', ] BR neobiker -- View this message in context: http://old.nabble.com/Xen-pv_ops-dom0-2.6.32.13-issues-tp28835895p28858517.html Sent from the Xen - Dev mailing list archive at Nabble.com. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: Which disk backend to use in domU? Date: Mon, 14 Jun 2010 11:49:45 +0100 Message-ID: <4C160949.7050007@goop.org> References: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> <4C1025C0.2070808@goop.org> <28857720.post@talk.nabble.com> <4C127584.6070801@dataproof.fi> <4C12781A.7060603@dataproof.fi> <28858517.post@talk.nabble.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <28858517.post@talk.nabble.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Neobiker Cc: xen-devel@lists.xensource.com, Daniel Stodden List-Id: xen-devel@lists.xenproject.org On 06/11/2010 07:11 PM, Neobiker wrote: > Hi > > Valtteri Kiviniemi-2 wrote: > >> Hi, Ah, misunderstanding sorry, you were talking about disk images :) >> >> > I'm talking about this config: > disk = [ > 'phy:/dev/vm/vm01,xvda1,w', > 'phy:/dev/vm/vm01-swap,xvda2,w', > 'phy:/dev/daten/devel_debian_amd64,xvda3,w', > ] > file: is definitely unsafe; its IO gets buffered in the dom0 pagecache, which means the guests writes aren't really writes. I believe phy: has similar problems, whereas tap:aio: implemented direct IO. But someone more storagey can confirm. J From mboxrd@z Thu Jan 1 00:00:00 1970 From: Daniel Stodden Subject: Re: Which disk backend to use in domU? Date: Mon, 14 Jun 2010 03:57:19 -0700 Message-ID: <1276513039.4191.6595.camel@ramone.somacoma.net> References: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> <4C1025C0.2070808@goop.org> <28857720.post@talk.nabble.com> <4C127584.6070801@dataproof.fi> <4C12781A.7060603@dataproof.fi> <28858517.post@talk.nabble.com> <4C160949.7050007@goop.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4C160949.7050007@goop.org> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Jeremy Fitzhardinge Cc: Neobiker , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org On Mon, 2010-06-14 at 06:49 -0400, Jeremy Fitzhardinge wrote: > On 06/11/2010 07:11 PM, Neobiker wrote: > > Hi > > > > Valtteri Kiviniemi-2 wrote: > > > >> Hi, Ah, misunderstanding sorry, you were talking about disk images :) > >> > >> > > I'm talking about this config: > > disk = [ > > 'phy:/dev/vm/vm01,xvda1,w', > > 'phy:/dev/vm/vm01-swap,xvda2,w', > > 'phy:/dev/daten/devel_debian_amd64,xvda3,w', > > ] > > > > file: is definitely unsafe; its IO gets buffered in the dom0 pagecache, > which means the guests writes aren't really writes. I believe phy: has > similar problems, whereas tap:aio: implemented direct IO. But someone > more storagey can confirm. Unless there's a difference in type names between XCP and .org, 'phy' means a bare LUN plugged into blkback? Those run underneath the entire block cache subsystems, which ironically has caching issues of it's own. But your writes are safe. Daniel From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pasi =?iso-8859-1?Q?K=E4rkk=E4inen?= Subject: Re: Which disk backend to use in domU? Date: Mon, 14 Jun 2010 14:01:19 +0300 Message-ID: <20100614110119.GB17817@reaktio.net> References: <1382341628.4833695.1276117331660.JavaMail.root@vms170009.mailsrvcs.net> <1742609046.4840453.1276126053830.JavaMail.root@vms170009.mailsrvcs.net> <4C1025C0.2070808@goop.org> <28857720.post@talk.nabble.com> <4C127584.6070801@dataproof.fi> <4C12781A.7060603@dataproof.fi> <28858517.post@talk.nabble.com> <4C160949.7050007@goop.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <4C160949.7050007@goop.org> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Jeremy Fitzhardinge Cc: Neobiker , xen-devel@lists.xensource.com, Daniel Stodden List-Id: xen-devel@lists.xenproject.org On Mon, Jun 14, 2010 at 11:49:45AM +0100, Jeremy Fitzhardinge wrote: > On 06/11/2010 07:11 PM, Neobiker wrote: > > Hi > > > > Valtteri Kiviniemi-2 wrote: > > > >> Hi, Ah, misunderstanding sorry, you were talking about disk images :) > >> > >> > > I'm talking about this config: > > disk = [ > > 'phy:/dev/vm/vm01,xvda1,w', > > 'phy:/dev/vm/vm01-swap,xvda2,w', > > 'phy:/dev/daten/devel_debian_amd64,xvda3,w', > > ] > > > > file: is definitely unsafe; its IO gets buffered in the dom0 pagecache, > which means the guests writes aren't really writes. I believe phy: has > similar problems, whereas tap:aio: implemented direct IO. But someone > more storagey can confirm. > I though phy: submits direct bio's bypassing dom0 pagecache.. -- Pasi