* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 [not found] ` <20100306132711.GK2580@reaktio.net> @ 2010-03-07 23:31 ` Josip Rodin 2010-03-08 23:16 ` Jeremy Fitzhardinge 2010-03-11 15:08 ` Josip Rodin 0 siblings, 2 replies; 18+ messages in thread From: Josip Rodin @ 2010-03-07 23:31 UTC (permalink / raw) To: Xen-devel On Sat, Mar 06, 2010 at 03:27:11PM +0200, Pasi Kärkkäinen wrote: > On Sat, Mar 06, 2010 at 12:58:33PM +0100, Josip Rodin wrote: > > I updated the hypervisor from 3.4.0 to 3.4.3rc3, which seemed to go well > > because it seems to behave as usual, but the dom0 kernel update from an > > earlier (pre-2009-11-30) xen/master snapshot based on 2.6.31.6 didn't - > > as soon as it starts, it just instantly reboots, or if noreboot is passed, > > it gets stuck, with absolutely nothing on the screen. > > So the 2009-11-30 xen/master 2.6.31.6 dom0 kernel works with Xen 3.4.3rc3 hypervisor, Yes. > but when you update the dom0 kernel to current xen/master 2.6.31.6, it crashes? When I update it to current xen/stable 2.6.32.9. (I put that bit in the subject only, sorry.) > > I tried adding loglvl=all guest_loglvl=all sync_console console_to_ring > > to the hypervisor options, and console=hvc0 earlyprintk=xen initcall_debug > > to the kernel options, to no avail. The hypervisor showed the sync console > > warning, but the dom0 kernel didn't change its behaviour. > > You should set up a serial console so you can capture the boot/error logs, > and we can analyze them to troubleshoot the problem, see: > > http://wiki.xensource.com/xenwiki/XenSerialConsole The machine has iLO VSP, so I'll try that next time I have a debugging time slot, thanks. -- 2. That which causes joy or happiness. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-07 23:31 ` current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 Josip Rodin @ 2010-03-08 23:16 ` Jeremy Fitzhardinge 2010-03-09 9:01 ` Josip Rodin 2010-03-11 15:08 ` Josip Rodin 1 sibling, 1 reply; 18+ messages in thread From: Jeremy Fitzhardinge @ 2010-03-08 23:16 UTC (permalink / raw) To: Josip Rodin; +Cc: Xen-devel On 03/07/2010 03:31 PM, Josip Rodin wrote: >> but when you update the dom0 kernel to current xen/master 2.6.31.6, it crashes? >> > When I update it to current xen/stable 2.6.32.9. (I put that bit in the > subject only, sorry.) > So 2.6.31.6 is OK on the 3.4.3rc3? It's just the new dom0 which fails? J ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-08 23:16 ` Jeremy Fitzhardinge @ 2010-03-09 9:01 ` Josip Rodin 2010-03-11 23:25 ` Yasir Assam 0 siblings, 1 reply; 18+ messages in thread From: Josip Rodin @ 2010-03-09 9:01 UTC (permalink / raw) To: Xen-devel On Mon, Mar 08, 2010 at 03:16:15PM -0800, Jeremy Fitzhardinge wrote: > On 03/07/2010 03:31 PM, Josip Rodin wrote: >>> but when you update the dom0 kernel to current xen/master 2.6.31.6, it crashes? >>> >> When I update it to current xen/stable 2.6.32.9. (I put that bit in the >> subject only, sorry.) > > So 2.6.31.6 is OK on the 3.4.3rc3? It's just the new dom0 which fails? Yes. (Sorry for the delay in further testing, I'm still waiting for users to ACK a new downtime.) -- 2. That which causes joy or happiness. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-09 9:01 ` Josip Rodin @ 2010-03-11 23:25 ` Yasir Assam 2010-03-11 23:40 ` Boris Derzhavets 0 siblings, 1 reply; 18+ messages in thread From: Yasir Assam @ 2010-03-11 23:25 UTC (permalink / raw) To: Xen-devel [-- Attachment #1.1: Type: text/plain, Size: 900 bytes --] I had this problem too a couple of days back (2.6.32.9 dom0 on Xen 3.4.2 - screen blank on boot). I didn't try 3.4.3rc3 but went straight to 4.0.0-rc6 and it booted ok. Yasir > On Mon, Mar 08, 2010 at 03:16:15PM -0800, Jeremy Fitzhardinge wrote: > >> On 03/07/2010 03:31 PM, Josip Rodin wrote: >> >>>> but when you update the dom0 kernel to current xen/master 2.6.31.6, it crashes? >>>> >>>> >>> When I update it to current xen/stable 2.6.32.9. (I put that bit in the >>> subject only, sorry.) >>> >> So 2.6.31.6 is OK on the 3.4.3rc3? It's just the new dom0 which fails? >> > Yes. > > (Sorry for the delay in further testing, I'm still waiting for users to > ACK a new downtime.) > > > > > > No virus found in this incoming message. > Checked by AVG - www.avg.com > Version: 9.0.733 / Virus Database: 271.1.1/2731 - Release Date: 03/09/10 06:33:00 > > [-- Attachment #1.2: Type: text/html, Size: 1577 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-11 23:25 ` Yasir Assam @ 2010-03-11 23:40 ` Boris Derzhavets 2010-03-12 2:22 ` Yasir Assam 0 siblings, 1 reply; 18+ messages in thread From: Boris Derzhavets @ 2010-03-11 23:40 UTC (permalink / raw) To: Xen-devel, Yasir Assam [-- Attachment #1.1: Type: text/plain, Size: 1436 bytes --] Try the most recent 3.4.3 and load 2.6.32.9 required CS back ported. --- On Thu, 3/11/10, Yasir Assam <mail@endlessvoid.com> wrote: From: Yasir Assam <mail@endlessvoid.com> Subject: Re: [Xen-devel] current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 To: "Xen-devel" <xen-devel@lists.xensource.com> Date: Thursday, March 11, 2010, 6:25 PM I had this problem too a couple of days back (2.6.32.9 dom0 on Xen 3.4.2 - screen blank on boot). I didn't try 3.4.3rc3 but went straight to 4.0.0-rc6 and it booted ok. Yasir On Mon, Mar 08, 2010 at 03:16:15PM -0800, Jeremy Fitzhardinge wrote: On 03/07/2010 03:31 PM, Josip Rodin wrote: but when you update the dom0 kernel to current xen/master 2.6.31.6, it crashes? When I update it to current xen/stable 2.6.32.9. (I put that bit in the subject only, sorry.) So 2.6.31.6 is OK on the 3.4.3rc3? It's just the new dom0 which fails? Yes. (Sorry for the delay in further testing, I'm still waiting for users to ACK a new downtime.) No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.733 / Virus Database: 271.1.1/2731 - Release Date: 03/09/10 06:33:00 -----Inline Attachment Follows----- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel [-- Attachment #1.2: Type: text/html, Size: 2413 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-11 23:40 ` Boris Derzhavets @ 2010-03-12 2:22 ` Yasir Assam 2010-03-12 7:14 ` Pasi Kärkkäinen 2010-03-12 11:28 ` Boris Derzhavets 0 siblings, 2 replies; 18+ messages in thread From: Yasir Assam @ 2010-03-12 2:22 UTC (permalink / raw) To: Boris Derzhavets; +Cc: Xen-devel [-- Attachment #1.1: Type: text/plain, Size: 1927 bytes --] I don't know what you mean by "load 2.6.32.9 required CS back ported" Yasir > Try the most recent 3.4.3 and load 2.6.32.9 required CS back ported. > > --- On *Thu, 3/11/10, Yasir Assam /<mail@endlessvoid.com>/* wrote: > > > From: Yasir Assam <mail@endlessvoid.com> > Subject: Re: [Xen-devel] current xen/stable 2.6.32.9 failed > upgrade from 2.6.31.6 > To: "Xen-devel" <xen-devel@lists.xensource.com> > Date: Thursday, March 11, 2010, 6:25 PM > > I had this problem too a couple of days back (2.6.32.9 dom0 on Xen > 3.4.2 - screen blank on boot). I didn't try 3.4.3rc3 but went > straight to 4.0.0-rc6 and it booted ok. > > Yasir >> On Mon, Mar 08, 2010 at 03:16:15PM -0800, Jeremy Fitzhardinge wrote: >> >>> On 03/07/2010 03:31 PM, Josip Rodin wrote: >>> >>>>> but when you update the dom0 kernel to current xen/master 2.6.31.6, it crashes? >>>>> >>>>> >>>> When I update it to current xen/stable 2.6.32.9. (I put that bit in the >>>> subject only, sorry.) >>>> >>> So 2.6.31.6 is OK on the 3.4.3rc3? It's just the new dom0 which fails? >>> >> Yes. >> >> (Sorry for the delay in further testing, I'm still waiting for users to >> ACK a new downtime.) >> >> >> >> >> >> No virus found in this incoming message. >> Checked by AVG -www.avg.com >> Version: 9.0.733 / Virus Database: 271.1.1/2731 - Release Date: 03/09/10 06:33:00 >> >> > > > -----Inline Attachment Follows----- > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > </mc/compose?to=Xen-devel@lists.xensource.com> > http://lists.xensource.com/xen-devel > > > > > No virus found in this incoming message. > Checked by AVG - www.avg.com > Version: 9.0.733 / Virus Database: 271.1.1/2736 - Release Date: 03/11/10 18:33:00 > > [-- Attachment #1.2: Type: text/html, Size: 3903 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-12 2:22 ` Yasir Assam @ 2010-03-12 7:14 ` Pasi Kärkkäinen 2010-03-12 11:28 ` Boris Derzhavets 1 sibling, 0 replies; 18+ messages in thread From: Pasi Kärkkäinen @ 2010-03-12 7:14 UTC (permalink / raw) To: Yasir Assam; +Cc: Boris Derzhavets, Xen-devel On Fri, Mar 12, 2010 at 01:22:31PM +1100, Yasir Assam wrote: > I don't know what you mean by "load 2.6.32.9 required CS back ported" > CS = ChangeSet. xen/stable 2.6.32.9 dom0 kernel requires Xen 3.4.3 (or a backport from 3.4.3 to 3.4.2 for the specific APIC hypercall CS), or Xen 4.0.0 hypervisor. -- Pasi > Yasir > > Try the most recent 3.4.3 and load 2.6.32.9 required CS back ported. > > --- On Thu, 3/11/10, Yasir Assam [1]<mail@endlessvoid.com> wrote: > > From: Yasir Assam [2]<mail@endlessvoid.com> > Subject: Re: [Xen-devel] current xen/stable 2.6.32.9 failed upgrade from > 2.6.31.6 > To: "Xen-devel" [3]<xen-devel@lists.xensource.com> > Date: Thursday, March 11, 2010, 6:25 PM > > I had this problem too a couple of days back (2.6.32.9 dom0 on Xen 3.4.2 - > screen blank on boot). I didn't try 3.4.3rc3 but went straight to 4.0.0-rc6 and > it booted ok. > > Yasir > > On Mon, Mar 08, 2010 at 03:16:15PM -0800, Jeremy Fitzhardinge wrote: > > > On 03/07/2010 03:31 PM, Josip Rodin wrote: > > > but when you update the dom0 kernel to current xen/master 2.6.31.6, it crashes? > > > > When I update it to current xen/stable 2.6.32.9. (I put that bit in the > subject only, sorry.) > > > So 2.6.31.6 is OK on the 3.4.3rc3? It's just the new dom0 which fails? > > > Yes. > > (Sorry for the delay in further testing, I'm still waiting for users to > ACK a new downtime.) > > > > > No virus found in this incoming message. > Checked by AVG - [4]www.avg.com > Version: 9.0.733 / Virus Database: 271.1.1/2731 - Release Date: 03/09/10 06:33:00 > > > > -----Inline Attachment Follows----- > > _______________________________________________ > Xen-devel mailing list > [5]Xen-devel@lists.xensource.com > [6]http://lists.xensource.com/xen-devel > > > > No virus found in this incoming message. > Checked by AVG - [7]www.avg.com > Version: 9.0.733 / Virus Database: 271.1.1/2736 - Release Date: 03/11/10 18:33:00 > > > > References > > Visible links > 1. mailto:mail@endlessvoid.com > 2. mailto:mail@endlessvoid.com > 3. mailto:xen-devel@lists.xensource.com > 4. http://www.avg.com/ > 5. file:///mc/compose?to=Xen-devel@lists.xensource.com > 6. http://lists.xensource.com/xen-devel > 7. http://www.avg.com/ > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-12 2:22 ` Yasir Assam 2010-03-12 7:14 ` Pasi Kärkkäinen @ 2010-03-12 11:28 ` Boris Derzhavets 2010-03-12 21:41 ` Yasir Assam 1 sibling, 1 reply; 18+ messages in thread From: Boris Derzhavets @ 2010-03-12 11:28 UTC (permalink / raw) To: Yasir Assam; +Cc: Xen-devel [-- Attachment #1.1: Type: text/plain, Size: 2933 bytes --] OK. Current 3.4.3 is good enough to load 2.6.32.9. You don't have upgrade Hypervisor to test 2.6.32.9. All required change sets (CSs) ( i guess just 1 by Xiantao Zang, which is connected with APIC rework ) have been back ported to 3.4.3 a while ago. Boris. --- On Thu, 3/11/10, Yasir Assam <mail@endlessvoid.com> wrote: From: Yasir Assam <mail@endlessvoid.com> Subject: Re: [Xen-devel] current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 To: "Boris Derzhavets" <bderzhavets@yahoo.com> Cc: "Xen-devel" <xen-devel@lists.xensource.com> Date: Thursday, March 11, 2010, 9:22 PM I don't know what you mean by "load 2.6.32.9 required CS back ported" Yasir Try the most recent 3.4.3 and load 2.6.32.9 required CS back ported. --- On Thu, 3/11/10, Yasir Assam <mail@endlessvoid.com> wrote: From: Yasir Assam <mail@endlessvoid.com> Subject: Re: [Xen-devel] current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 To: "Xen-devel" <xen-devel@lists.xensource.com> Date: Thursday, March 11, 2010, 6:25 PM I had this problem too a couple of days back (2.6.32.9 dom0 on Xen 3.4.2 - screen blank on boot). I didn't try 3.4.3rc3 but went straight to 4.0.0-rc6 and it booted ok. Yasir On Mon, Mar 08, 2010 at 03:16:15PM -0800, Jeremy Fitzhardinge wrote: On 03/07/2010 03:31 PM, Josip Rodin wrote: but when you update the dom0 kernel to current xen/master 2.6.31.6, it crashes? When I update it to current xen/stable 2.6.32.9. (I put that bit in the subject only, sorry.) So 2.6.31.6 is OK on the 3.4.3rc3? It's just the new dom0 which fails? Yes. (Sorry for the delay in further testing, I'm still waiting for users to ACK a new downtime.) No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.733 / Virus Database: 271.1.1/2731 - Release Date: 03/09/10 06:33:00 -----Inline Attachment Follows----- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.733 / Virus Database: 271.1.1/2736 - Release Date: 03/11/10 18:33:00 -----Inline Attachment Follows----- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel [-- Attachment #1.2: Type: text/html, Size: 5206 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-12 11:28 ` Boris Derzhavets @ 2010-03-12 21:41 ` Yasir Assam 0 siblings, 0 replies; 18+ messages in thread From: Yasir Assam @ 2010-03-12 21:41 UTC (permalink / raw) To: Boris Derzhavets; +Cc: Xen-devel [-- Attachment #1.1: Type: text/plain, Size: 3543 bytes --] I'm going to stick with 4.0.0rc6 for now because it seems to be stable - this is a production machine (hardware is too new for older stable dom0 & xen which is why I'm using the latest software). Yasir > OK. Current 3.4.3 is good enough to load 2.6.32.9. You don't have upgrade > Hypervisor to test 2.6.32.9. All required change sets (CSs) ( i guess > just 1 by Xiantao Zang, which is connected with APIC rework ) have > been back ported to 3.4.3 a while ago. > > Boris. > > --- On *Thu, 3/11/10, Yasir Assam /<mail@endlessvoid.com>/* wrote: > > > From: Yasir Assam <mail@endlessvoid.com> > Subject: Re: [Xen-devel] current xen/stable 2.6.32.9 failed > upgrade from 2.6.31.6 > To: "Boris Derzhavets" <bderzhavets@yahoo.com> > Cc: "Xen-devel" <xen-devel@lists.xensource.com> > Date: Thursday, March 11, 2010, 9:22 PM > > I don't know what you mean by "load 2.6.32.9 required CS back ported" > > Yasir >> Try the most recent 3.4.3 and load 2.6.32.9 required CS back ported. >> >> --- On *Thu, 3/11/10, Yasir Assam /<mail@endlessvoid.com>/* wrote: >> >> >> From: Yasir Assam <mail@endlessvoid.com> >> Subject: Re: [Xen-devel] current xen/stable 2.6.32.9 failed >> upgrade from 2.6.31.6 >> To: "Xen-devel" <xen-devel@lists.xensource.com> >> Date: Thursday, March 11, 2010, 6:25 PM >> >> I had this problem too a couple of days back (2.6.32.9 dom0 >> on Xen 3.4.2 - screen blank on boot). I didn't try 3.4.3rc3 >> but went straight to 4.0.0-rc6 and it booted ok. >> >> Yasir >>> On Mon, Mar 08, 2010 at 03:16:15PM -0800, Jeremy Fitzhardinge wrote: >>> >>>> On 03/07/2010 03:31 PM, Josip Rodin wrote: >>>> >>>>>> but when you update the dom0 kernel to current xen/master 2.6.31.6, it crashes? >>>>>> >>>>>> >>>>> When I update it to current xen/stable 2.6.32.9. (I put that bit in the >>>>> subject only, sorry.) >>>>> >>>> So 2.6.31.6 is OK on the 3.4.3rc3? It's just the new dom0 which fails? >>>> >>> Yes. >>> >>> (Sorry for the delay in further testing, I'm still waiting for users to >>> ACK a new downtime.) >>> >>> >>> >>> >>> >>> No virus found in this incoming message. >>> Checked by AVG -www.avg.com >>> Version: 9.0.733 / Virus Database: 271.1.1/2731 - Release Date: 03/09/10 06:33:00 >>> >>> >> >> >> -----Inline Attachment Follows----- >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> >> >> >> >> No virus found in this incoming message. >> Checked by AVG -www.avg.com >> Version: 9.0.733 / Virus Database: 271.1.1/2736 - Release Date: 03/11/10 18:33:00 >> >> > > > -----Inline Attachment Follows----- > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > </mc/compose?to=Xen-devel@lists.xensource.com> > http://lists.xensource.com/xen-devel > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > > > > No virus found in this incoming message. > Checked by AVG - www.avg.com > Version: 9.0.733 / Virus Database: 271.1.1/2736 - Release Date: 03/11/10 18:33:00 > > [-- Attachment #1.2: Type: text/html, Size: 7617 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-07 23:31 ` current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 Josip Rodin 2010-03-08 23:16 ` Jeremy Fitzhardinge @ 2010-03-11 15:08 ` Josip Rodin 2010-03-11 19:24 ` Pasi Kärkkäinen 1 sibling, 1 reply; 18+ messages in thread From: Josip Rodin @ 2010-03-11 15:08 UTC (permalink / raw) To: Xen-devel On Mon, Mar 08, 2010 at 12:31:47AM +0100, Josip Rodin wrote: > > > I tried adding loglvl=all guest_loglvl=all sync_console console_to_ring > > > to the hypervisor options, and console=hvc0 earlyprintk=xen initcall_debug > > > to the kernel options, to no avail. The hypervisor showed the sync console > > > warning, but the dom0 kernel didn't change its behaviour. > > > > You should set up a serial console so you can capture the boot/error logs, > > and we can analyze them to troubleshoot the problem, see: > > > > http://wiki.xensource.com/xenwiki/XenSerialConsole > > The machine has iLO VSP, so I'll try that next time I have a debugging > time slot, thanks. Sadly, it looks this iLO VSP is completely useless, because when I defined com1, the hypervisor started *not* displaying anything on the physical console or the virtual serial port. I tried this once and it made the machine seem broken for a while, when all of the sudden fifteen minutes later it happily showed the getty on ttyS1 on VSP. I wonder if perhaps this was caused by some hypervisor console lock timeout expiring, so I tried to change the setup a bit by removing that getty from inittab. Then I rebooted it again, and the machine again became catatonic for a while, but it didn't come back like the last time - it doesn't ping, nothing is shown on physical console, and nothing is shown on VSP. :( Maybe I'm missing something regarding the iLO VSP setup on this HP DL380... -- 2. That which causes joy or happiness. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-11 15:08 ` Josip Rodin @ 2010-03-11 19:24 ` Pasi Kärkkäinen 2010-03-12 11:41 ` Josip Rodin 0 siblings, 1 reply; 18+ messages in thread From: Pasi Kärkkäinen @ 2010-03-11 19:24 UTC (permalink / raw) To: Josip Rodin; +Cc: Xen-devel On Thu, Mar 11, 2010 at 04:08:23PM +0100, Josip Rodin wrote: > On Mon, Mar 08, 2010 at 12:31:47AM +0100, Josip Rodin wrote: > > > > I tried adding loglvl=all guest_loglvl=all sync_console console_to_ring > > > > to the hypervisor options, and console=hvc0 earlyprintk=xen initcall_debug > > > > to the kernel options, to no avail. The hypervisor showed the sync console > > > > warning, but the dom0 kernel didn't change its behaviour. > > > > > > You should set up a serial console so you can capture the boot/error logs, > > > and we can analyze them to troubleshoot the problem, see: > > > > > > http://wiki.xensource.com/xenwiki/XenSerialConsole > > > > The machine has iLO VSP, so I'll try that next time I have a debugging > > time slot, thanks. > > Sadly, it looks this iLO VSP is completely useless, because when I defined > com1, the hypervisor started *not* displaying anything on the physical > console or the virtual serial port. I tried this once and it made the > machine seem broken for a while, when all of the sudden fifteen minutes > later it happily showed the getty on ttyS1 on VSP. > > I wonder if perhaps this was caused by some hypervisor console lock timeout > expiring, so I tried to change the setup a bit by removing that getty from > inittab. Then I rebooted it again, and the machine again became catatonic > for a while, but it didn't come back like the last time - it doesn't ping, > nothing is shown on physical console, and nothing is shown on VSP. :( > > Maybe I'm missing something regarding the iLO VSP setup on this HP DL380... > If you boot baremetal (non-Xen) Linux, on what IO-port and IRQ is the iLO serial port? Is it on the standard IO-port/IRQ, or some non-standard? In this case you might have to specify the IOport on the xen cmdline options in grub.conf. See also: http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00263709/c00263709.pdf -- Pasi ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-11 19:24 ` Pasi Kärkkäinen @ 2010-03-12 11:41 ` Josip Rodin 2010-03-12 12:09 ` Josip Rodin 2010-03-12 13:02 ` Pasi Kärkkäinen 0 siblings, 2 replies; 18+ messages in thread From: Josip Rodin @ 2010-03-12 11:41 UTC (permalink / raw) To: Xen-devel On Thu, Mar 11, 2010 at 09:24:56PM +0200, Pasi Kärkkäinen wrote: > > Sadly, it looks this iLO VSP is completely useless, because when I defined > > com1, the hypervisor started *not* displaying anything on the physical > > console or the virtual serial port. > > Maybe I'm missing something regarding the iLO VSP setup on this HP DL380... > > If you boot baremetal (non-Xen) Linux, on what IO-port and IRQ is the iLO > serial port? > > Is it on the standard IO-port/IRQ, or some non-standard? In this case you > might have to specify the IOport on the xen cmdline options in grub.conf. > > See also: > http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00263709/c00263709.pdf Oh, thanks, that was useful. The manual actually documents that VSP is on COM *2*, not 1, and despite my prior ample negative experience with the off-by-one in port numbering, and the fact I had even used ttyS1=COM2 on this exact machine, I managed to screw that one up. Sorry for the noise. So, I managed to get GRUB, the hypervisor and dom0 to display messages on VSP. (No luck yet with BIOS POST, but that is irrelevant for now.) I'll first paste the few odd bits of log from the *working* 2.6.31.6 dom0, just to set up a possible baseline: [...] [ 0.148467] PCI: Using ACPI for IRQ routing [ 0.152000] pci 0000:00:1d.0: BAR 4: address space collision on of device [0x2000-0x201f] [ 0.152000] pci 0000:00:1d.0: BAR 4: can't allocate resource [ 0.152000] pci 0000:00:1d.1: BAR 4: address space collision on of device [0x2020-0x203f] [ 0.152000] pci 0000:00:1d.1: BAR 4: can't allocate resource [ 0.152000] pci 0000:00:1d.2: BAR 4: address space collision on of device [0x2040-0x205f] [ 0.152000] pci 0000:00:1d.2: BAR 4: can't allocate resource [ 0.152000] pci 0000:00:1d.3: BAR 4: address space collision on of device [0x2060-0x207f] [ 0.152000] pci 0000:00:1d.3: BAR 4: can't allocate resource [ 0.152000] IO APIC resources couldn't be allocated. [...] [ 1.603357] xen_set_ioapic_routing: irq 25 gsi 25 vector 160 ioapic 1 pin 1 triggering 1 polarity 1 [ 1.603377] tg3 0000:03:01.0: PCI INT A -> GSI 25 (level, low) -> IRQ 25 [ 1.604933] uhci_hcd: USB Universal Host Controller Interface driver [ 1.605036] xen_allocate_pirq: returning irq 16 for gsi 16 [ 1.605047] xen_set_ioapic_routing: irq 16 gsi 16 vector 144 ioapic 0 pin 16 triggering 1 polarity 1 [ 1.605060] uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 [ 1.605088] uhci_hcd 0000:00:1d.0: UHCI Host Controller [ 1.605204] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1 [ 1.605295] uhci_hcd 0000:00:1d.0: irq 16, io base 0x00008000 [ 1.605429] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.605438] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.605445] usb usb1: Product: UHCI Host Controller [ 1.605450] usb usb1: Manufacturer: Linux 2.6.31.6 uhci_hcd [ 1.605456] usb usb1: SerialNumber: 0000:00:1d.0 [ 1.605534] usb usb1: configuration #1 chosen from 1 choice [ 1.605588] hub 1-0:1.0: USB hub found [ 1.605617] hub 1-0:1.0: 2 ports detected [ 1.605774] xen_set_ioapic_routing: irq 19 gsi 19 vector 168 ioapic 0 pin 19 triggering 1 polarity 1 [ 1.605789] uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19 [ 1.605813] uhci_hcd 0000:00:1d.1: UHCI Host Controller [ 1.605871] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2 [ 1.605950] uhci_hcd 0000:00:1d.1: irq 19, io base 0x00008020 [ 1.606084] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.606092] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.606098] usb usb2: Product: UHCI Host Controller [ 1.606104] usb usb2: Manufacturer: Linux 2.6.31.6 uhci_hcd [ 1.606109] usb usb2: SerialNumber: 0000:00:1d.1 [ 1.606183] usb usb2: configuration #1 chosen from 1 choice [ 1.606255] hub 2-0:1.0: USB hub found [ 1.606283] hub 2-0:1.0: 2 ports detected [ 1.606418] xen_set_ioapic_routing: irq 18 gsi 18 vector 176 ioapic 0 pin 18 triggering 1 polarity 1 [ 1.606429] uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18 [ 1.606453] uhci_hcd 0000:00:1d.2: UHCI Host Controller [ 1.606514] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3 [ 1.606588] uhci_hcd 0000:00:1d.2: irq 18, io base 0x00008040 [ 1.606696] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.606703] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.606710] usb usb3: Product: UHCI Host Controller [ 1.606716] usb usb3: Manufacturer: Linux 2.6.31.6 uhci_hcd [ 1.606721] usb usb3: SerialNumber: 0000:00:1d.2 [ 1.606793] usb usb3: configuration #1 chosen from 1 choice [ 1.606846] hub 3-0:1.0: USB hub found [ 1.606874] hub 3-0:1.0: 2 ports detected [ 1.607004] xen_allocate_pirq: returning irq 16 for gsi 16 [ 1.607020] xen_set_ioapic_routing: irq 16 gsi 16 vector 144 ioapic 0 pin 16 triggering 1 polarity 1 (XEN) io_apic.c:2139: (XEN) ioapic_guest_write: apic=0, pin=16, old_irq=16, new_irq=16 (XEN) ioapic_guest_write: old_entry=0000a990, new_entry=0001a990 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! [ 1.607048] uhci_hcd 0000:00:1d.3: PCI INT A -> GSI 16 (level, low) -> IRQ 16 [ 1.607080] uhci_hcd 0000:00:1d.3: UHCI Host Controller [ 1.607146] uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 4 [ 1.607206] uhci_hcd 0000:00:1d.3: irq 16, io base 0x00008060 [ 1.607312] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001 [ 1.607320] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 1.607327] usb usb4: Product: UHCI Host Controller [ 1.607332] usb usb4: Manufacturer: Linux 2.6.31.6 uhci_hcd [ 1.607338] usb usb4: SerialNumber: 0000:00:1d.3 [ 1.607407] usb usb4: configuration #1 chosen from 1 choice [ 1.607457] hub 4-0:1.0: USB hub found [ 1.607486] hub 4-0:1.0: 2 ports detected [...] Starting Xen daemons: xenstored xenconsoled xend[ 44.446762] xenbus_probe wake_waiting [ 44.446792] xenbus_probe wake_waiting [ 44.447309] XENBUS: Unable to read cpu state [ 44.447586] XENBUS: Unable to read cpu state [ 44.447858] XENBUS: Unable to read cpu state [ 44.448416] XENBUS: Unable to read cpu state [ 44.448851] xenbus_probe_devices backend [ 44.459623] xenbus_probe_devices failed xenbus_directory [ 44.459635] backend_probe_and_watch devices probed ok [ 44.459794] backend_probe_and_watch watch add ok ok [ 44.459800] backend_probe_and_watch all done [ 44.459806] xenbus_probe_devices device [ 44.459973] xenbus_probe_devices failed xenbus_directory [ 44.459980] frontend_probe_and_watch devices probed ok [ 44.460557] frontend_probe_and_watch watch add ok ok [ 44.460565] frontend_probe_and_watch all done . Everything else seems normal, and indeed the machine has worked normally in this setup for a few months. And now here goes the whole output preceding the 2.6.32 crash: (XEN) Xen version 3.4 (Debian 3.4.3~rc3-1) (waldi@debian.org) (gcc version 4.4.3 (Debian 4.4.3-2) ) Mon Mar 1 21:16:18 UTC 2010 (XEN) Command line: dom0_mem=512M com2=115200,8n1 console=com2 noreboot loglvl=all guest_loglvl=all (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) VBE/DDC methods: V2; EDID transfer time: 2 seconds (XEN) Disc information: (XEN) Found 1 MBR signatures (XEN) Found 1 EDD information structures (XEN) Xen-e820 RAM map: (XEN) 0000000000000000 - 000000000009f400 (usable) (XEN) 000000000009f400 - 00000000000a0000 (reserved) (XEN) 00000000000f0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 00000000dfff3000 (usable) (XEN) 00000000dfff3000 - 00000000dfffb000 (ACPI data) (XEN) 00000000dfffb000 - 00000000e0000000 (reserved) (XEN) 00000000fec00000 - 00000000fed00000 (reserved) (XEN) 00000000fee00000 - 00000000fee10000 (reserved) (XEN) 00000000ffc00000 - 0000000100000000 (reserved) (XEN) 0000000100000000 - 000000011bfff000 (usable) (XEN) System RAM: 4031MB (4128324kB) (XEN) ACPI: RSDP 000F4F00, 0024 (r2 HP ) (XEN) ACPI: XSDT DFFF3300, 004C (r1 HP P51 2 162E) (XEN) ACPI: FACP DFFF3380, 00F4 (r3 HP P51 2 162E) (XEN) ACPI: DSDT DFFF3480, 2BC3 (r1 HP DSDT 1 INTL 20030228) (XEN) ACPI: FACS DFFF3100, 0040 (XEN) ACPI: SPCR DFFF3140, 0050 (r1 HP SPCRRBSU 1 162E) (XEN) ACPI: MCFG DFFF31C0, 003C (r1 HP ProLiant 1 0) (XEN) ACPI: APIC DFFF3200, 00C2 (r1 HP 00000083 2 0) (XEN) ACPI: SSDT DFFF8000, 02CD (r1 HP SSDTP 1 INTL 20030228) (XEN) NUMA turned off (XEN) Faking a node at 0000000000000000-000000011bfff000 (XEN) Domain heap initialised (XEN) found SMP MP-table at 000f4f80 (XEN) DMI 2.3 present. (XEN) Using APIC driver default (XEN) ACPI: PM-Timer IO Port: 0x908 (XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[904,0], pm1x_evt[900,0] (XEN) ACPI: wakeup_vec[dfff310c], vec_size[20] (XEN) ACPI: Local APIC address 0xfee00000 (XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) (XEN) Processor #0 15:4 APIC version 20 (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled) (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled) (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) (XEN) Processor #6 15:4 APIC version 20 (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) (XEN) Processor #1 15:4 APIC version 20 (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled) (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled) (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) (XEN) Processor #7 15:4 APIC version 20 (XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) (XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) (XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 (XEN) ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24]) (XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47 (XEN) ACPI: IOAPIC (id[0x0a] address[0xfec80400] gsi_base[48]) (XEN) IOAPIC[2]: apic_id 10, version 32, address 0xfec80400, GSI 48-71 (XEN) ACPI: IOAPIC (id[0x0b] address[0xfec84000] gsi_base[72]) (XEN) IOAPIC[3]: apic_id 11, version 32, address 0xfec84000, GSI 72-95 (XEN) ACPI: IOAPIC (id[0x0c] address[0xfec84400] gsi_base[96]) (XEN) IOAPIC[4]: apic_id 12, version 32, address 0xfec84400, GSI 96-119 (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) (XEN) ACPI: IRQ0 used by override. (XEN) ACPI: IRQ2 used by override. (XEN) ACPI: IRQ9 used by override. (XEN) Enabling APIC mode: Flat. Using 5 I/O APICs (XEN) Using ACPI (MADT) for SMP configuration information (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Initializing CPU#0 (XEN) Detected 3400.282 MHz processor. (XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K (XEN) CPU: L2 cache: 2048K (XEN) CPU: Physical Processor ID: 0 (XEN) CPU0: Intel Extended MCE MSRs (24) available (XEN) Intel machine check reporting enabled on CPU#0. (XEN) CPU0: Thermal monitoring enabled (TM1) (XEN) CMCI: CPU0 has no CMCI support (XEN) CPU0: Intel(R) Xeon(TM) CPU 3.40GHz stepping 03 (XEN) Booting processor 1/6 eip 88000 (XEN) Initializing CPU#1 (XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K (XEN) CPU: L2 cache: 2048K (XEN) CPU: Physical Processor ID: 3 (XEN) CPU1: Intel Extended MCE MSRs (24) available (XEN) Intel machine check reporting enabled on CPU#1. (XEN) CPU1: Thermal monitoring enabled (TM1) (XEN) CMCI: CPU1 has no CMCI support (XEN) CPU1: Intel(R) Xeon(TM) CPU 3.40GHz stepping 03 (XEN) Booting processor 2/1 eip 88000 (XEN) Initializing CPU#2 (XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K (XEN) CPU: L2 cache: 2048K (XEN) CPU: Physical Processor ID: 0 (XEN) CPU2: Intel Extended MCE MSRs (24) available (XEN) Intel machine check reporting enabled on CPU#2. (XEN) CPU2: Thermal monitoring enabled (TM1) (XEN) CMCI: CPU2 has no CMCI support (XEN) CPU2: Intel(R) Xeon(TM) CPU 3.40GHz stepping 03 (XEN) Booting processor 3/7 eip 88000 (XEN) Initializing CPU#3 (XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K (XEN) CPU: L2 cache: 2048K (XEN) CPU: Physical Processor ID: 3 (XEN) CPU3: Intel Extended MCE MSRs (24) available (XEN) Intel machine check reporting enabled on CPU#3. (XEN) CPU3: Thermal monitoring enabled (TM1) (XEN) CMCI: CPU3 has no CMCI support (XEN) CPU3: Intel(R) Xeon(TM) CPU 3.40GHz stepping 03 (XEN) Total of 4 processors activated. (XEN) ENABLING IO-APIC IRQs (XEN) -> Using new ACK method (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) checking TSC synchronization across 4 CPUs: passed. (XEN) Platform timer is 3.579MHz ACPI PM Timer (XEN) microcode.c:73:d32767 microcode: CPU1 resumed (XEN) microcode.c:73:d32767 microcode: CPU3 resumed (XEN) Brought up 4 CPUs (XEN) microcode.c:73:d32767 microcode: CPU2 resumed (XEN) I/O virtualisation disabled (XEN) CPUIDLE: disabled due to no HPET. Force enable with 'cpuidle'. (XEN) ACPI sleep modes: S3 (XEN) mcheck_poll: Machine check polling timer started. (XEN) mtrr: your CPUs had inconsistent fixed MTRR settings (XEN) mtrr: probably your BIOS does not setup all CPUs. (XEN) mtrr: corrected configuration. (XEN) *** LOADING DOMAIN 0 *** (XEN) Xen kernel: 64-bit, lsb, compat32 (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x15fb000 (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 0000000114000000->0000000118000000 (114688 pages to be allocated) (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: ffffffff81000000->ffffffff815fb000 (XEN) Init. ramdisk: ffffffff815fb000->ffffffff82b7b600 (XEN) Phys-Mach map: ffffffff82b7c000->ffffffff82c7c000 (XEN) Start info: ffffffff82c7c000->ffffffff82c7c4b4 (XEN) Page tables: ffffffff82c7d000->ffffffff82c98000 (XEN) Boot stack: ffffffff82c98000->ffffffff82c99000 (XEN) TOTAL: ffffffff80000000->ffffffff83000000 (XEN) ENTRY ADDRESS: ffffffff81472200 (XEN) Dom0 has maximum 4 VCPUs (XEN) Scrubbing Free RAM: ..................................done. (XEN) Xen trace buffers: disabled (XEN) Std. Loglevel: All (XEN) Guest Loglevel: All (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) Freed 132kB init memory. mapping kernel into physical memory Xen: setup ISA identity maps about to get started... [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 2.6.32.9 (joy@virgo) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #7 SMP Thu Mar 4 23:40:34 CET 2010 [ 0.000000] Command line: root=LABEL=virgo-root ro console=hvc0 earlyprintk=xen nomodeset [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] Centaur CentaurHauls [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] Xen: 0000000000000000 - 000000000009f400 (usable) [ 0.000000] Xen: 000000000009f400 - 0000000000100000 (reserved) [ 0.000000] Xen: 0000000000100000 - 0000000020000000 (usable) [ 0.000000] Xen: 00000000dfff3000 - 00000000dfffb000 (ACPI data) [ 0.000000] Xen: 00000000dfffb000 - 00000000e0000000 (reserved) [ 0.000000] Xen: 00000000fec00000 - 00000000fed00000 (reserved) [ 0.000000] Xen: 00000000fee00000 - 00000000fee10000 (reserved) [ 0.000000] Xen: 00000000ffc00000 - 0000000100000000 (reserved) [ 0.000000] bootconsole [xenboot0] enabled [ 0.000000] DMI 2.3 present. [ 0.000000] last_pfn = 0x20000 max_arch_pfn = 0x400000000 [ 0.000000] init_memory_mapping: 0000000000000000-0000000020000000 [ 0.000000] RAMDISK: 015fb000 - 02b7b600 [ 0.000000] ACPI: RSDP 00000000000f4f00 00024 (v02 HP ) [ 0.000000] ACPI: XSDT 00000000dfff3300 0004C (v01 HP P51 00000002 ? 0000162E) [ 0.000000] ACPI: FACP 00000000dfff3380 000F4 (v03 HP P51 00000002 ? 0000162E) [ 0.000000] ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607) [ 0.000000] ACPI: DSDT 00000000dfff3480 02BC3 (v01 HP DSDT 00000001 INTL 20030228) [ 0.000000] ACPI: FACS 00000000dfff3100 00040 [ 0.000000] ACPI: SPCR 00000000dfff3140 00050 (v01 HP SPCRRBSU 00000001 ? 0000162E) [ 0.000000] ACPI: MCFG 00000000dfff31c0 0003C (v01 HP ProLiant 00000001 00000000) [ 0.000000] ACPI: APIC 00000000dfff3200 000C2 (v01 HP 00000083 00000002 00000000) [ 0.000000] ACPI: SSDT 00000000dfff8000 002CD (v01 HP SSDTP 00000001 INTL 20030228) [ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 0020000000] [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] [ 0.000000] #1 [0002c7d000 - 0002c98000] XEN PAGETABLES ==> [0002c7d000 - 0002c98000] [ 0.000000] #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] [ 0.000000] #3 [0001000000 - 00015dadd4] TEXT DATA BSS ==> [0001000000 - 00015dadd4] [ 0.000000] #4 [00015fb000 - 0002b7b600] RAMDISK ==> [00015fb000 - 0002b7b600] [ 0.000000] #5 [0002b7c000 - 0002c7d000] XEN START INFO ==> [0002b7c000 - 0002c7d000] [ 0.000000] #6 [00015db000 - 00015db0d4] BRK ==> [00015db000 - 00015db0d4] [ 0.000000] #7 [0000100000 - 00001e4000] PGTABLE ==> [0000100000 - 00001e4000] [ 0.000000] found SMP MP-table at [ffff8800000f4f80] f4f80 [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000000 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal 0x00100000 -> 0x00100000 [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000000 -> 0x0000009f [ 0.000000] 0: 0x00000100 -> 0x00020000 [ 0.000000] ACPI: PM-Timer IO Port: 0x908 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) (XEN) mm.c:720:d0 Bad L1 flags 800000 (XEN) mm.c:4221:d0 ptwr_emulate: could not get_page_from_l1e() (XEN) d0:v0: unhandled page fault (ec=0003) (XEN) Pagetable walk from ffff8800014fdfd8: (XEN) L4[0x110] = 0000000115002067 0000000000001002 (XEN) L3[0x000] = 0000000115006067 0000000000001006 (XEN) L2[0x00a] = 0000000116c8a067 0000000000002c8a (XEN) L1[0x0fd] = 00100001154fd065 00000000000014fd (XEN) domain_crash_sync called from entry.S (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-3.4 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<ffffffff8100d9df>] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (XEN) rax: 0000000000000000 rbx: 80000001154fc163 rcx: 00000000014fd000 (XEN) rdx: 0000000000000000 rsi: 80000001154fc163 rdi: ffff8800014fdfd8 (XEN) rbp: ffff8800014fdfd8 rsp: ffffffff813abd70 r8: 0000000000000002 (XEN) r9: ffffffff8146a060 r10: ffffffff8100b6a3 r11: ffffffff8146a060 (XEN) r12: 80000001154fc163 r13: 0000000000000000 r14: ffffffffff4002c2 (XEN) r15: ffffffff8147cbb0 cr0: 000000008005003b cr4: 00000000000006f0 (XEN) cr3: 0000000115001000 cr2: ffff8800014fdfd8 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff813abd70: (XEN) 00000000014fd000 ffffffff8146a060 0000000000000003 ffffffff8100d9df (XEN) 000000010000e030 0000000000010046 ffffffff813abdb8 000000000000e02b (XEN) ffffffff8100d9df ffffffffff5fb000 ffffffffff5fb000 0000000000000804 (XEN) ffffffff81025db1 0000000115003067 80000001154fc163 ffffffff81001000 (XEN) 80000001154fc163 0000000000000008 ffffffff81029e63 0000000000000000 (XEN) ffffffff8100c40d ffffffff813abe88 0000000000000000 0000000000000000 (XEN) ffffffff814816a0 ffffffff8147cbb0 ffffffffff40026c ffffffffff40026c (XEN) 0000000000000080 ffffffff81336a73 ffffffff8147cbdc 0000000000000001 (XEN) ffffffff814925aa 00000001813abf78 00000000000000c2 ffffffffff400200 (XEN) 0000000000000000 0000000000000000 ffffffffffffffff ffffffff813abf78 (XEN) 0000000000000000 0000000000000000 ffffffff8147d220 ffffffff813abf78 (XEN) 0000000001580600 0000000020000000 ffffffff81477685 ffffffff8100db3d (XEN) 0000000002b7b600 00000000015fb000 0000000002b7b600 ffffffff8133b055 (XEN) ffffffff8133fb05 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 ffffffffffffffff ffffffff814a2090 (XEN) 0000000000000000 0000000000000000 ffffffff8147297d 00000000015dadd4 (XEN) 0000000000000000 ffffffff814a3fc0 ffffffff81001000 0000000000000000 (XEN) 0000000000000000 0000000000000000 ffffffff81475686 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000001 (XEN) Domain 0 crashed: 'noreboot' set - not rebooting. In the meantime there was another update to the stable branch, I'll go compile that... -- 2. That which causes joy or happiness. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-12 11:41 ` Josip Rodin @ 2010-03-12 12:09 ` Josip Rodin 2010-03-23 23:18 ` Josip Rodin 2010-03-12 13:02 ` Pasi Kärkkäinen 1 sibling, 1 reply; 18+ messages in thread From: Josip Rodin @ 2010-03-12 12:09 UTC (permalink / raw) To: Xen-devel On Fri, Mar 12, 2010 at 12:41:39PM +0100, Josip Rodin wrote: > And now here goes the whole output preceding the 2.6.32 crash: [...] > In the meantime there was another update to the stable branch, I'll go > compile that... The symptoms remained the same, only the CPU MHz calculation and some memory offsets are different. (XEN) Xen version 3.4 (Debian 3.4.3~rc3-1) (waldi@debian.org) (gcc version 4.4.3 (Debian 4.4.3-2) ) Mon Mar 1 21:16:18 UTC 2010 (XEN) Command line: dom0_mem=512M com2=115200,8n1 console=com2 noreboot loglvl=all guest_loglvl=all (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) VBE/DDC methods: V2; EDID transfer time: 2 seconds (XEN) Disc information: (XEN) Found 1 MBR signatures (XEN) Found 1 EDD information structures (XEN) Xen-e820 RAM map: (XEN) 0000000000000000 - 000000000009f400 (usable) (XEN) 000000000009f400 - 00000000000a0000 (reserved) (XEN) 00000000000f0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 00000000dfff3000 (usable) (XEN) 00000000dfff3000 - 00000000dfffb000 (ACPI data) (XEN) 00000000dfffb000 - 00000000e0000000 (reserved) (XEN) 00000000fec00000 - 00000000fed00000 (reserved) (XEN) 00000000fee00000 - 00000000fee10000 (reserved) (XEN) 00000000ffc00000 - 0000000100000000 (reserved) (XEN) 0000000100000000 - 000000011bfff000 (usable) (XEN) System RAM: 4031MB (4128324kB) (XEN) ACPI: RSDP 000F4F00, 0024 (r2 HP ) (XEN) ACPI: XSDT DFFF3300, 004C (r1 HP P51 2 162E) (XEN) ACPI: FACP DFFF3380, 00F4 (r3 HP P51 2 162E) (XEN) ACPI: DSDT DFFF3480, 2BC3 (r1 HP DSDT 1 INTL 20030228) (XEN) ACPI: FACS DFFF3100, 0040 (XEN) ACPI: SPCR DFFF3140, 0050 (r1 HP SPCRRBSU 1 162E) (XEN) ACPI: MCFG DFFF31C0, 003C (r1 HP ProLiant 1 0) (XEN) ACPI: APIC DFFF3200, 00C2 (r1 HP 00000083 2 0) (XEN) ACPI: SSDT DFFF8000, 02CD (r1 HP SSDTP 1 INTL 20030228) (XEN) NUMA turned off (XEN) Faking a node at 0000000000000000-000000011bfff000 (XEN) Domain heap initialised (XEN) found SMP MP-table at 000f4f80 (XEN) DMI 2.3 present. (XEN) Using APIC driver default (XEN) ACPI: PM-Timer IO Port: 0x908 (XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[904,0], pm1x_evt[900,0] (XEN) ACPI: wakeup_vec[dfff310c], vec_size[20] (XEN) ACPI: Local APIC address 0xfee00000 (XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) (XEN) Processor #0 15:4 APIC version 20 (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled) (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled) (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) (XEN) Processor #6 15:4 APIC version 20 (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) (XEN) Processor #1 15:4 APIC version 20 (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled) (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled) (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) (XEN) Processor #7 15:4 APIC version 20 (XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) (XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) (XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 (XEN) ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24]) (XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47 (XEN) ACPI: IOAPIC (id[0x0a] address[0xfec80400] gsi_base[48]) (XEN) IOAPIC[2]: apic_id 10, version 32, address 0xfec80400, GSI 48-71 (XEN) ACPI: IOAPIC (id[0x0b] address[0xfec84000] gsi_base[72]) (XEN) IOAPIC[3]: apic_id 11, version 32, address 0xfec84000, GSI 72-95 (XEN) ACPI: IOAPIC (id[0x0c] address[0xfec84400] gsi_base[96]) (XEN) IOAPIC[4]: apic_id 12, version 32, address 0xfec84400, GSI 96-119 (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) (XEN) ACPI: IRQ0 used by override. (XEN) ACPI: IRQ2 used by override. (XEN) ACPI: IRQ9 used by override. (XEN) Enabling APIC mode: Flat. Using 5 I/O APICs (XEN) Using ACPI (MADT) for SMP configuration information (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Initializing CPU#0 (XEN) Detected 3400.333 MHz processor. (XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K (XEN) CPU: L2 cache: 2048K (XEN) CPU: Physical Processor ID: 0 (XEN) CPU0: Intel Extended MCE MSRs (24) available (XEN) Intel machine check reporting enabled on CPU#0. (XEN) CPU0: Thermal monitoring enabled (TM1) (XEN) CMCI: CPU0 has no CMCI support (XEN) CPU0: Intel(R) Xeon(TM) CPU 3.40GHz stepping 03 (XEN) Booting processor 1/6 eip 88000 (XEN) Initializing CPU#1 (XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K (XEN) CPU: L2 cache: 2048K (XEN) CPU: Physical Processor ID: 3 (XEN) CPU1: Intel Extended MCE MSRs (24) available (XEN) Intel machine check reporting enabled on CPU#1. (XEN) CPU1: Thermal monitoring enabled (TM1) (XEN) CMCI: CPU1 has no CMCI support (XEN) CPU1: Intel(R) Xeon(TM) CPU 3.40GHz stepping 03 (XEN) Booting processor 2/1 eip 88000 (XEN) Initializing CPU#2 (XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K (XEN) CPU: L2 cache: 2048K (XEN) CPU: Physical Processor ID: 0 (XEN) CPU2: Intel Extended MCE MSRs (24) available (XEN) Intel machine check reporting enabled on CPU#2. (XEN) CPU2: Thermal monitoring enabled (TM1) (XEN) CMCI: CPU2 has no CMCI support (XEN) CPU2: Intel(R) Xeon(TM) CPU 3.40GHz stepping 03 (XEN) Booting processor 3/7 eip 88000 (XEN) Initializing CPU#3 (XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K (XEN) CPU: L2 cache: 2048K (XEN) CPU: Physical Processor ID: 3 (XEN) CPU3: Intel Extended MCE MSRs (24) available (XEN) Intel machine check reporting enabled on CPU#3. (XEN) CPU3: Thermal monitoring enabled (TM1) (XEN) CMCI: CPU3 has no CMCI support (XEN) CPU3: Intel(R) Xeon(TM) CPU 3.40GHz stepping 03 (XEN) Total of 4 processors activated. (XEN) ENABLING IO-APIC IRQs (XEN) -> Using new ACK method (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) checking TSC synchronization across 4 CPUs: passed. (XEN) Platform timer is 3.579MHz ACPI PM Timer (XEN) microcode.c:73:d32767 microcode: CPU1 resumed (XEN) microcode.c:73:d32767 microcode: CPU3 resumed (XEN) microcode.c:73:d32767 microcode: CPU2 resumed (XEN) Brought up 4 CPUs (XEN) I/O virtualisation disabled (XEN) CPUIDLE: disabled due to no HPET. Force enable with 'cpuidle'. (XEN) ACPI sleep modes: S3 (XEN) mcheck_poll: Machine check polling timer started. (XEN) mtrr: your CPUs had inconsistent fixed MTRR settings (XEN) mtrr: probably your BIOS does not setup all CPUs. (XEN) mtrr: corrected configuration. (XEN) *** LOADING DOMAIN 0 *** (XEN) Xen kernel: 64-bit, lsb, compat32 (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x15fb000 (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 0000000114000000->0000000118000000 (114688 pages to be allocated) (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: ffffffff81000000->ffffffff815fb000 (XEN) Init. ramdisk: ffffffff815fb000->ffffffff82b7b600 (XEN) Phys-Mach map: ffffffff82b7c000->ffffffff82c7c000 (XEN) Start info: ffffffff82c7c000->ffffffff82c7c4b4 (XEN) Page tables: ffffffff82c7d000->ffffffff82c98000 (XEN) Boot stack: ffffffff82c98000->ffffffff82c99000 (XEN) TOTAL: ffffffff80000000->ffffffff83000000 (XEN) ENTRY ADDRESS: ffffffff81472200 (XEN) Dom0 has maximum 4 VCPUs (XEN) Scrubbing Free RAM: ..................................done. (XEN) Xen trace buffers: disabled (XEN) Std. Loglevel: All (XEN) Guest Loglevel: All (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) Freed 132kB init memory. mapping kernel into physical memory Xen: setup ISA identity maps about to get started... [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 2.6.32.9 (joy@virgo) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #8 SMP Fri Mar 12 12:57:42 CET 2010 [ 0.000000] Command line: root=LABEL=virgo-root ro console=hvc0 earlyprintk=xen nomodeset [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] Centaur CentaurHauls [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] Xen: 0000000000000000 - 000000000009f400 (usable) [ 0.000000] Xen: 000000000009f400 - 0000000000100000 (reserved) [ 0.000000] Xen: 0000000000100000 - 0000000020000000 (usable) [ 0.000000] Xen: 00000000dfff3000 - 00000000dfffb000 (ACPI data) [ 0.000000] Xen: 00000000dfffb000 - 00000000e0000000 (reserved) [ 0.000000] Xen: 00000000fec00000 - 00000000fed00000 (reserved) [ 0.000000] Xen: 00000000fee00000 - 00000000fee10000 (reserved) [ 0.000000] Xen: 00000000ffc00000 - 0000000100000000 (reserved) [ 0.000000] bootconsole [xenboot0] enabled [ 0.000000] DMI 2.3 present. [ 0.000000] last_pfn = 0x20000 max_arch_pfn = 0x400000000 [ 0.000000] init_memory_mapping: 0000000000000000-0000000020000000 [ 0.000000] RAMDISK: 015fb000 - 02b7b600 [ 0.000000] ACPI: RSDP 00000000000f4f00 00024 (v02 HP ) [ 0.000000] ACPI: XSDT 00000000dfff3300 0004C (v01 HP P51 00000002 ? 0000162E) [ 0.000000] ACPI: FACP 00000000dfff3380 000F4 (v03 HP P51 00000002 ? 0000162E) [ 0.000000] ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20090903/tbfadt-607) [ 0.000000] ACPI: DSDT 00000000dfff3480 02BC3 (v01 HP DSDT 00000001 INTL 20030228) [ 0.000000] ACPI: FACS 00000000dfff3100 00040 [ 0.000000] ACPI: SPCR 00000000dfff3140 00050 (v01 HP SPCRRBSU 00000001 ? 0000162E) [ 0.000000] ACPI: MCFG 00000000dfff31c0 0003C (v01 HP ProLiant 00000001 00000000) [ 0.000000] ACPI: APIC 00000000dfff3200 000C2 (v01 HP 00000083 00000002 00000000) [ 0.000000] ACPI: SSDT 00000000dfff8000 002CD (v01 HP SSDTP 00000001 INTL 20030228) [ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 0020000000] [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] [ 0.000000] #1 [0002c7d000 - 0002c98000] XEN PAGETABLES ==> [0002c7d000 - 0002c98000] [ 0.000000] #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] [ 0.000000] #3 [0001000000 - 00015dae14] TEXT DATA BSS ==> [0001000000 - 00015dae14] [ 0.000000] #4 [00015fb000 - 0002b7b600] RAMDISK ==> [00015fb000 - 0002b7b600] [ 0.000000] #5 [0002b7c000 - 0002c7d000] XEN START INFO ==> [0002b7c000 - 0002c7d000] [ 0.000000] #6 [00015db000 - 00015db0d4] BRK ==> [00015db000 - 00015db0d4] [ 0.000000] #7 [0000100000 - 00001e4000] PGTABLE ==> [0000100000 - 00001e4000] [ 0.000000] found SMP MP-table at [ffff8800000f4f80] f4f80 [ 0.000000] Zone PFN ranges: [ 0.000000] DMA 0x00000000 -> 0x00001000 [ 0.000000] DMA32 0x00001000 -> 0x00100000 [ 0.000000] Normal 0x00100000 -> 0x00100000 [ 0.000000] Movable zone start PFN for each node [ 0.000000] early_node_map[2] active PFN ranges [ 0.000000] 0: 0x00000000 -> 0x0000009f [ 0.000000] 0: 0x00000100 -> 0x00020000 [ 0.000000] ACPI: PM-Timer IO Port: 0x908 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) (XEN) mm.c:720:d0 Bad L1 flags 800000 (XEN) mm.c:4221:d0 ptwr_emulate: could not get_page_from_l1e() (XEN) d0:v0: unhandled page fault (ec=0003) (XEN) Pagetable walk from ffff8800014fdfd8: (XEN) L4[0x110] = 0000000115002067 0000000000001002 (XEN) L3[0x000] = 0000000115006067 0000000000001006 (XEN) L2[0x00a] = 0000000116c8a067 0000000000002c8a (XEN) L1[0x0fd] = 00100001154fd065 00000000000014fd (XEN) domain_crash_sync called from entry.S (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-3.4 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e033:[<ffffffff8100d9df>] (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest (XEN) rax: 0000000000000000 rbx: 80000001154fc163 rcx: 00000000014fd000 (XEN) rdx: 0000000000000000 rsi: 80000001154fc163 rdi: ffff8800014fdfd8 (XEN) rbp: ffff8800014fdfd8 rsp: ffffffff813abd70 r8: 0000000000000002 (XEN) r9: ffffffff8146a060 r10: ffffffff8100b6a3 r11: ffffffff8146a060 (XEN) r12: 80000001154fc163 r13: 0000000000000000 r14: ffffffffff4002c2 (XEN) r15: ffffffff8147cbb0 cr0: 000000008005003b cr4: 00000000000006f0 (XEN) cr3: 0000000115001000 cr2: ffff8800014fdfd8 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff813abd70: (XEN) 00000000014fd000 ffffffff8146a060 0000000000000003 ffffffff8100d9df (XEN) 000000010000e030 0000000000010046 ffffffff813abdb8 000000000000e02b (XEN) ffffffff8100d9df ffffffffff5fb000 ffffffffff5fb000 0000000000000804 (XEN) ffffffff81025e2d 0000000115003067 80000001154fc163 ffffffff81001000 (XEN) 80000001154fc163 0000000000000008 ffffffff81029edf 0000000000000000 (XEN) ffffffff8100c40d ffffffff813abe88 0000000000000000 0000000000000000 (XEN) ffffffff814816a0 ffffffff8147cbb0 ffffffffff40026c ffffffffff40026c (XEN) 0000000000000080 ffffffff81336ae3 ffffffff8147cbdc 0000000000000001 (XEN) ffffffff814925aa 00000001813abf78 00000000000000c2 ffffffffff400200 (XEN) 0000000000000000 0000000000000000 ffffffffffffffff ffffffff813abf78 (XEN) 0000000000000000 0000000000000000 ffffffff8147d220 ffffffff813abf78 (XEN) 0000000001580600 0000000020000000 ffffffff81477685 ffffffff8100db3d (XEN) 0000000002b7b600 00000000015fb000 0000000002b7b600 ffffffff8133b0c5 (XEN) ffffffff8133fb75 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 ffffffffffffffff ffffffff814a2090 (XEN) 0000000000000000 0000000000000000 ffffffff8147297d 00000000015dae14 (XEN) 0000000000000000 ffffffff814a3fc0 ffffffff81001000 0000000000000000 (XEN) 0000000000000000 0000000000000000 ffffffff81475686 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000001 (XEN) Domain 0 crashed: 'noreboot' set - not rebooting. JFTR that's with git commit 1181ebba81ef9143338c18f0d1c767a5ab407e05 as HEAD -- 2. That which causes joy or happiness. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-12 12:09 ` Josip Rodin @ 2010-03-23 23:18 ` Josip Rodin 2010-03-23 23:22 ` Josip Rodin 0 siblings, 1 reply; 18+ messages in thread From: Josip Rodin @ 2010-03-23 23:18 UTC (permalink / raw) To: Xen-devel On Fri, Mar 12, 2010 at 01:09:14PM +0100, Josip Rodin wrote: > On Fri, Mar 12, 2010 at 12:41:39PM +0100, Josip Rodin wrote: > > And now here goes the whole output preceding the 2.6.32 crash: > [...] > > In the meantime there was another update to the stable branch, I'll go > > compile that... > > The symptoms remained the same, only the CPU MHz calculation and some memory > offsets are different. > > (XEN) mm.c:720:d0 Bad L1 flags 800000 > (XEN) mm.c:4221:d0 ptwr_emulate: could not get_page_from_l1e() > (XEN) d0:v0: unhandled page fault (ec=0003) > (XEN) Pagetable walk from ffff8800014fdfd8: > (XEN) L4[0x110] = 0000000115002067 0000000000001002 > (XEN) L3[0x000] = 0000000115006067 0000000000001006 > (XEN) L2[0x00a] = 0000000116c8a067 0000000000002c8a > (XEN) L1[0x0fd] = 00100001154fd065 00000000000014fd > (XEN) domain_crash_sync called from entry.S > (XEN) Domain 0 (vcpu#0) crashed on cpu#0: > (XEN) ----[ Xen-3.4 x86_64 debug=n Not tainted ]---- FWIW I tried to git bisect this in the last couple of days, but the result turned out to be fairly obvious and useless as after 14 bisections I only came to this: commit 18ecfad3aaeead019b0e07078f643deaa7d10d44 x86: make /dev/mem mappings _PAGE_IOMAP commit 56f27a6d47275f6dc94adf3ecc5fe958cdcdebee xen/dom0: add XEN_DOM0 config option I didn't follow through with the last bisection, it had seemed increasingly futile for a while now... :) I saw a peculiar side effect at one point, when I went back to a random working 2.6.31.1 dom0, all userland processes started crashing with Illegal instruction. One iLO reset later, it's all good again. I'm guessing it was a transient broken state. And then when I gave up and updated to latest xen/stable for one last try, that was the biggest d'oh moment - it's fixed :) Was it de67ec8b? -- 2. That which causes joy or happiness. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-23 23:18 ` Josip Rodin @ 2010-03-23 23:22 ` Josip Rodin 2010-03-24 5:10 ` Jeremy Fitzhardinge 0 siblings, 1 reply; 18+ messages in thread From: Josip Rodin @ 2010-03-23 23:22 UTC (permalink / raw) To: Xen-devel On Wed, Mar 24, 2010 at 12:18:53AM +0100, Josip Rodin wrote: > On Fri, Mar 12, 2010 at 01:09:14PM +0100, Josip Rodin wrote: > > On Fri, Mar 12, 2010 at 12:41:39PM +0100, Josip Rodin wrote: > > > And now here goes the whole output preceding the 2.6.32 crash: > > [...] > > > In the meantime there was another update to the stable branch, I'll go > > > compile that... > > > > The symptoms remained the same, only the CPU MHz calculation and some memory > > offsets are different. > > > > (XEN) mm.c:720:d0 Bad L1 flags 800000 > > (XEN) mm.c:4221:d0 ptwr_emulate: could not get_page_from_l1e() > > (XEN) d0:v0: unhandled page fault (ec=0003) > > (XEN) Pagetable walk from ffff8800014fdfd8: > > (XEN) L4[0x110] = 0000000115002067 0000000000001002 > > (XEN) L3[0x000] = 0000000115006067 0000000000001006 > > (XEN) L2[0x00a] = 0000000116c8a067 0000000000002c8a > > (XEN) L1[0x0fd] = 00100001154fd065 00000000000014fd > > (XEN) domain_crash_sync called from entry.S > > (XEN) Domain 0 (vcpu#0) crashed on cpu#0: > > (XEN) ----[ Xen-3.4 x86_64 debug=n Not tainted ]---- > > FWIW I tried to git bisect this in the last couple of days, but the result > turned out to be fairly obvious and useless as after 14 bisections > I only came to this: > > commit 18ecfad3aaeead019b0e07078f643deaa7d10d44 > x86: make /dev/mem mappings _PAGE_IOMAP > commit 56f27a6d47275f6dc94adf3ecc5fe958cdcdebee > xen/dom0: add XEN_DOM0 config option > > I didn't follow through with the last bisection, it had seemed increasingly > futile for a while now... :) > > I saw a peculiar side effect at one point, when I went back to a random > working 2.6.31.1 dom0, all userland processes started crashing with Illegal > instruction. One iLO reset later, it's all good again. I'm guessing it was > a transient broken state. > > And then when I gave up and updated to latest xen/stable for one last try, > that was the biggest d'oh moment - it's fixed :) Was it de67ec8b? BTW with the working .32 kernel, the log says: [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 8, version 0, address 0xfec00000, GSI 0-0 [ 0.000000] ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 9, version 0, address 0xfec80000, GSI 24-24 [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec80400] gsi_base[48]) [ 0.000000] IOAPIC[2]: apic_id 10, version 0, address 0xfec80400, GSI 48-48 [ 0.000000] ACPI: IOAPIC (id[0x0b] address[0xfec84000] gsi_base[72]) [ 0.000000] IOAPIC[3]: apic_id 11, version 0, address 0xfec84000, GSI 72-72 [ 0.000000] ACPI: IOAPIC (id[0x0c] address[0xfec84400] gsi_base[96]) [ 0.000000] IOAPIC[4]: apic_id 12, version 0, address 0xfec84400, GSI 96-96 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) [ 0.000000] ERROR: Unable to locate IOAPIC for GSI 2 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ERROR: Unable to locate IOAPIC for GSI 9 [...] [ 0.023694] ACPI: bus type pci registered [ 0.023915] PCI: Found Intel Corporation E7520 Memory Controller Hub with MMCONFIG support. [ 0.023935] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255 [ 0.023942] PCI: Not using MMCONFIG. [ 0.023948] PCI: Using configuration type 1 for base access [ 0.023959] PCI: HP ProLiant DL380 detected, enabling pci=bfsort. [ 0.028634] bio: create slab <bio-0> at 0 [ 0.030115] ERROR: Unable to locate IOAPIC for GSI 9 Is there anything I can do to avoid these? -- Josip Rodin Racunalno-informacijski sustavi i servisi CARNet - Croatian Academic and Research Network J. Marohnica 5, 10000 Zagreb, Croatia tel. +385 1 66 61 61 6 http://www.carnet.hr/ ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-23 23:22 ` Josip Rodin @ 2010-03-24 5:10 ` Jeremy Fitzhardinge 2010-03-24 8:37 ` Josip Rodin 0 siblings, 1 reply; 18+ messages in thread From: Jeremy Fitzhardinge @ 2010-03-24 5:10 UTC (permalink / raw) To: Josip Rodin; +Cc: Xen-devel On 03/23/2010 04:22 PM, Josip Rodin wrote: > On Wed, Mar 24, 2010 at 12:18:53AM +0100, Josip Rodin wrote: > >> On Fri, Mar 12, 2010 at 01:09:14PM +0100, Josip Rodin wrote: >> >>> On Fri, Mar 12, 2010 at 12:41:39PM +0100, Josip Rodin wrote: >>> >>>> And now here goes the whole output preceding the 2.6.32 crash: >>>> >>> [...] >>> >>>> In the meantime there was another update to the stable branch, I'll go >>>> compile that... >>>> >>> The symptoms remained the same, only the CPU MHz calculation and some memory >>> offsets are different. >>> >>> (XEN) mm.c:720:d0 Bad L1 flags 800000 >>> (XEN) mm.c:4221:d0 ptwr_emulate: could not get_page_from_l1e() >>> (XEN) d0:v0: unhandled page fault (ec=0003) >>> (XEN) Pagetable walk from ffff8800014fdfd8: >>> (XEN) L4[0x110] = 0000000115002067 0000000000001002 >>> (XEN) L3[0x000] = 0000000115006067 0000000000001006 >>> (XEN) L2[0x00a] = 0000000116c8a067 0000000000002c8a >>> (XEN) L1[0x0fd] = 00100001154fd065 00000000000014fd >>> (XEN) domain_crash_sync called from entry.S >>> (XEN) Domain 0 (vcpu#0) crashed on cpu#0: >>> (XEN) ----[ Xen-3.4 x86_64 debug=n Not tainted ]---- >>> >> FWIW I tried to git bisect this in the last couple of days, but the result >> turned out to be fairly obvious and useless as after 14 bisections >> I only came to this: >> >> commit 18ecfad3aaeead019b0e07078f643deaa7d10d44 >> x86: make /dev/mem mappings _PAGE_IOMAP >> commit 56f27a6d47275f6dc94adf3ecc5fe958cdcdebee >> xen/dom0: add XEN_DOM0 config option >> >> I didn't follow through with the last bisection, it had seemed increasingly >> futile for a while now... :) >> >> I saw a peculiar side effect at one point, when I went back to a random >> working 2.6.31.1 dom0, all userland processes started crashing with Illegal >> instruction. One iLO reset later, it's all good again. I'm guessing it was >> a transient broken state. >> >> And then when I gave up and updated to latest xen/stable for one last try, >> that was the biggest d'oh moment - it's fixed :) Was it de67ec8b? >> Yes. > BTW with the working .32 kernel, the log says: > > [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) > [ 0.000000] IOAPIC[0]: apic_id 8, version 0, address 0xfec00000, GSI 0-0 > [ 0.000000] ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24]) > [ 0.000000] IOAPIC[1]: apic_id 9, version 0, address 0xfec80000, GSI 24-24 > [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec80400] gsi_base[48]) > [ 0.000000] IOAPIC[2]: apic_id 10, version 0, address 0xfec80400, GSI 48-48 > [ 0.000000] ACPI: IOAPIC (id[0x0b] address[0xfec84000] gsi_base[72]) > [ 0.000000] IOAPIC[3]: apic_id 11, version 0, address 0xfec84000, GSI 72-72 > [ 0.000000] ACPI: IOAPIC (id[0x0c] address[0xfec84400] gsi_base[96]) > [ 0.000000] IOAPIC[4]: apic_id 12, version 0, address 0xfec84400, GSI 96-96 > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) > [ 0.000000] ERROR: Unable to locate IOAPIC for GSI 2 > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) > [ 0.000000] ERROR: Unable to locate IOAPIC for GSI 9 > > [...] > > [ 0.023694] ACPI: bus type pci registered > [ 0.023915] PCI: Found Intel Corporation E7520 Memory Controller Hub with MMCONFIG support. > [ 0.023935] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255 > [ 0.023942] PCI: Not using MMCONFIG. > [ 0.023948] PCI: Using configuration type 1 for base access > [ 0.023959] PCI: HP ProLiant DL380 detected, enabling pci=bfsort. > [ 0.028634] bio: create slab<bio-0> at 0 > [ 0.030115] ERROR: Unable to locate IOAPIC for GSI 9 > > Is there anything I can do to avoid these? > These are just noise; the kernel thinks it can poke at the IO APICs, but they're owned by Xen and so don't exist for the kernel; instead some alternate mechanisms come into play to keep the interrupts flowing. At some point I hope we can completely remove all trace of the APICs from the kernel's sight, so it won't even try to access them and print these confused messages. J ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-24 5:10 ` Jeremy Fitzhardinge @ 2010-03-24 8:37 ` Josip Rodin 0 siblings, 0 replies; 18+ messages in thread From: Josip Rodin @ 2010-03-24 8:37 UTC (permalink / raw) To: Jeremy Fitzhardinge; +Cc: Xen-devel On Tue, Mar 23, 2010 at 10:10:48PM -0700, Jeremy Fitzhardinge wrote: >>> And then when I gave up and updated to latest xen/stable for one last try, >>> that was the biggest d'oh moment - it's fixed :) Was it de67ec8b? > > Yes. OK. Thanks. :) In the meantime I've verified that the fixed kernel also works on another 32-bit server (a Dell PE750, 2003) where it had also previously crashed on boot. >> [ 0.000000] ERROR: Unable to locate IOAPIC for GSI 2 >> [ 0.000000] ERROR: Unable to locate IOAPIC for GSI 9 >> [ 0.030115] ERROR: Unable to locate IOAPIC for GSI 9 >> >> Is there anything I can do to avoid these? > > These are just noise; the kernel thinks it can poke at the IO APICs, but > they're owned by Xen and so don't exist for the kernel; instead some > alternate mechanisms come into play to keep the interrupts flowing. At > some point I hope we can completely remove all trace of the APICs from > the kernel's sight, so it won't even try to access them and print these > confused messages. OK. I just noticed it because it trips up the default check_dmesg Nagios verification, as it has the word 'error' in it. -- 2. That which causes joy or happiness. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 2010-03-12 11:41 ` Josip Rodin 2010-03-12 12:09 ` Josip Rodin @ 2010-03-12 13:02 ` Pasi Kärkkäinen 1 sibling, 0 replies; 18+ messages in thread From: Pasi Kärkkäinen @ 2010-03-12 13:02 UTC (permalink / raw) To: Josip Rodin; +Cc: Xen-devel On Fri, Mar 12, 2010 at 12:41:39PM +0100, Josip Rodin wrote: > On Thu, Mar 11, 2010 at 09:24:56PM +0200, Pasi Kärkkäinen wrote: > > > Sadly, it looks this iLO VSP is completely useless, because when I defined > > > com1, the hypervisor started *not* displaying anything on the physical > > > console or the virtual serial port. > > > Maybe I'm missing something regarding the iLO VSP setup on this HP DL380... > > > > If you boot baremetal (non-Xen) Linux, on what IO-port and IRQ is the iLO > > serial port? > > > > Is it on the standard IO-port/IRQ, or some non-standard? In this case you > > might have to specify the IOport on the xen cmdline options in grub.conf. > > > > See also: > > http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00263709/c00263709.pdf > > Oh, thanks, that was useful. The manual actually documents that VSP is on > COM *2*, not 1, and despite my prior ample negative experience with the > off-by-one in port numbering, and the fact I had even used ttyS1=COM2 on > this exact machine, I managed to screw that one up. Sorry for the noise. > > So, I managed to get GRUB, the hypervisor and dom0 to display messages on > VSP. (No luck yet with BIOS POST, but that is irrelevant for now.) > > I'll first paste the few odd bits of log from the *working* 2.6.31.6 dom0, > just to set up a possible baseline: > <snip> > > Everything else seems normal, and indeed the machine has worked normally > in this setup for a few months. > > And now here goes the whole output preceding the 2.6.32 crash: > <snip> > > [ 0.000000] ACPI: PM-Timer IO Port: 0x908 > [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled) > [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) > [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) > [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) > (XEN) mm.c:720:d0 Bad L1 flags 800000 > (XEN) mm.c:4221:d0 ptwr_emulate: could not get_page_from_l1e() > (XEN) d0:v0: unhandled page fault (ec=0003) > (XEN) Pagetable walk from ffff8800014fdfd8: > (XEN) L4[0x110] = 0000000115002067 0000000000001002 > (XEN) L3[0x000] = 0000000115006067 0000000000001006 > (XEN) L2[0x00a] = 0000000116c8a067 0000000000002c8a > (XEN) L1[0x0fd] = 00100001154fd065 00000000000014fd > (XEN) domain_crash_sync called from entry.S > (XEN) Domain 0 (vcpu#0) crashed on cpu#0: > (XEN) ----[ Xen-3.4 x86_64 debug=n Not tainted ]---- > (XEN) CPU: 0 > (XEN) RIP: e033:[<ffffffff8100d9df>] > (XEN) RFLAGS: 0000000000000246 EM: 1 CONTEXT: pv guest > (XEN) rax: 0000000000000000 rbx: 80000001154fc163 rcx: 00000000014fd000 > (XEN) rdx: 0000000000000000 rsi: 80000001154fc163 rdi: ffff8800014fdfd8 > (XEN) rbp: ffff8800014fdfd8 rsp: ffffffff813abd70 r8: 0000000000000002 > (XEN) r9: ffffffff8146a060 r10: ffffffff8100b6a3 r11: ffffffff8146a060 > (XEN) r12: 80000001154fc163 r13: 0000000000000000 r14: ffffffffff4002c2 > (XEN) r15: ffffffff8147cbb0 cr0: 000000008005003b cr4: 00000000000006f0 > (XEN) cr3: 0000000115001000 cr2: ffff8800014fdfd8 > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 > (XEN) Guest stack trace from rsp=ffffffff813abd70: > (XEN) 00000000014fd000 ffffffff8146a060 0000000000000003 ffffffff8100d9df > (XEN) 000000010000e030 0000000000010046 ffffffff813abdb8 000000000000e02b > (XEN) ffffffff8100d9df ffffffffff5fb000 ffffffffff5fb000 0000000000000804 > (XEN) ffffffff81025db1 0000000115003067 80000001154fc163 ffffffff81001000 > (XEN) 80000001154fc163 0000000000000008 ffffffff81029e63 0000000000000000 > (XEN) ffffffff8100c40d ffffffff813abe88 0000000000000000 0000000000000000 > (XEN) ffffffff814816a0 ffffffff8147cbb0 ffffffffff40026c ffffffffff40026c > (XEN) 0000000000000080 ffffffff81336a73 ffffffff8147cbdc 0000000000000001 > (XEN) ffffffff814925aa 00000001813abf78 00000000000000c2 ffffffffff400200 > (XEN) 0000000000000000 0000000000000000 ffffffffffffffff ffffffff813abf78 > (XEN) 0000000000000000 0000000000000000 ffffffff8147d220 ffffffff813abf78 > (XEN) 0000000001580600 0000000020000000 ffffffff81477685 ffffffff8100db3d > (XEN) 0000000002b7b600 00000000015fb000 0000000002b7b600 ffffffff8133b055 > (XEN) ffffffff8133fb05 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 ffffffffffffffff ffffffff814a2090 > (XEN) 0000000000000000 0000000000000000 ffffffff8147297d 00000000015dadd4 > (XEN) 0000000000000000 ffffffff814a3fc0 ffffffff81001000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 ffffffff81475686 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000001 > (XEN) Domain 0 crashed: 'noreboot' set - not rebooting. > > In the meantime there was another update to the stable branch, I'll go > compile that... > This is a known problem.. I see it also on my old 32bit testbox. 64bit dom0 kernel works for me. Jeremy has some ideas about what might be causing this.. -- Pasi ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2010-03-24 8:37 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20100306115833.GA28039@orion.carnet.hr>
[not found] ` <20100306132711.GK2580@reaktio.net>
2010-03-07 23:31 ` current xen/stable 2.6.32.9 failed upgrade from 2.6.31.6 Josip Rodin
2010-03-08 23:16 ` Jeremy Fitzhardinge
2010-03-09 9:01 ` Josip Rodin
2010-03-11 23:25 ` Yasir Assam
2010-03-11 23:40 ` Boris Derzhavets
2010-03-12 2:22 ` Yasir Assam
2010-03-12 7:14 ` Pasi Kärkkäinen
2010-03-12 11:28 ` Boris Derzhavets
2010-03-12 21:41 ` Yasir Assam
2010-03-11 15:08 ` Josip Rodin
2010-03-11 19:24 ` Pasi Kärkkäinen
2010-03-12 11:41 ` Josip Rodin
2010-03-12 12:09 ` Josip Rodin
2010-03-23 23:18 ` Josip Rodin
2010-03-23 23:22 ` Josip Rodin
2010-03-24 5:10 ` Jeremy Fitzhardinge
2010-03-24 8:37 ` Josip Rodin
2010-03-12 13:02 ` Pasi Kärkkäinen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).