From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ian Campbell Subject: Re: osstest going offline for a bit due to database server move Date: Mon, 16 Mar 2015 12:41:23 +0000 Message-ID: <1426509683.18247.45.camel@citrix.com> References: <21760.27967.26101.11897@mariner.uk.xensource.com> <1426330945.28029.73.camel@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1YXUKm-0004JU-72 for xen-devel@lists.xenproject.org; Mon, 16 Mar 2015 12:41:28 +0000 In-Reply-To: <1426330945.28029.73.camel@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Ian Jackson Cc: xen-devel@lists.xenproject.org, Jan Beulich List-Id: xen-devel@lists.xenproject.org On Sat, 2015-03-14 at 11:02 +0000, Ian Campbell wrote: > On Wed, 2015-03-11 at 16:28 +0000, Ian Jackson wrote: > > The db server for the production osstest instance needs to be > > physically moved. This is planned to take place on Friday. > > > > I have dropped a `stop' file in which will stop osstest taking on new > > work. The db server will be moved on Friday morning (and anything > > still running then will be killed). > > > > We expect to resume service some time on Friday. > > TL;DR: The move took place successfully (?) but osstest is still > unavailable. :-/ > > Since the move of the DB server there seems to have been some issues > with the (unrelated) filer which supplies storage to the server hosting > the controller VM which is causing the VM's rootfs to go read only about > once a day. I've rebooted it again this morning but TBH based on recent > history I don't expect it to survive until tomorrow. > > Separately there seems to be some sort of new issue with the PDU control > software (which may or may not relate to the db move since PDU control > is via the DB), so when the VM is up and running it isn't able to > actually do much which is useful. > > I'll be looking into both of those with some urgency on Monday since > there isn't much I can do from here. In the meantime I don't think there > will be much in the way of useful test results. The PDU issue is fixed. We've not yet tracked down the source of the mysterious filer reboots and there was another earlier today, we've fiddled with a few things to see if we can track them down. osstest is doing stuff now, fingers crossed. Ian.