Xen Source Code Overview Document

Xen Source Code Overview Document

[Stephen Spector]

[-- Attachment #1.1: Type: text/plain, Size: 1316 bytes --]

Xen Developers:

The response from the community survey indicated that people were looking for a Xen source code overview document that would help them find particular files based on the feature they were interested in. To support this, I have created a document (see attached in .odt and .doc) that lists every file in the xen source tree as well as a brief description of the content in each file; where I could understand the code).  If you know what is in a file that has no text next to it; please let me know and I will update the documents.

The next step of this process is to write a brief description of each directory to help people find the code they need to examine.  I need some help here.  If just a few developers took 10 minutes a day to write a brief description of 1 or 2 directories a day I could have this document completed by the end of the week.

Thanks for your help in finishing this document.

PS - I have also posted a new Configuration document that lists all the possible Domain configuration file settings; this document is in review on xen-users.

Stephen Spector
Community Manager, Xen.org
stephen.spector@xen.org<mailto:stephen.spector@xen.org>
(772) 621-5062
BLOG: http://blog.xen.org<http://blog.xen.org/>
Folllow Me on Twitter: http://twitter.com/xen_com_mgr



[-- Attachment #1.2: Type: text/html, Size: 6771 bytes --]

[-- Attachment #2: xen source guide_v1.doc --]
[-- Type: application/msword, Size: 199680 bytes --]

[-- Attachment #3: xen source guide_v1.odt --]
[-- Type: application/octet-stream, Size: 44256 bytes --]

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

What is the current state of Dom0 kernel support?

[Anthony Wright]

I've seen a number of posts relating to dom0 support in more recent
kernel versions, but I'm unsure what the current situation is and
wondered if somebody might be able clarify things.

I saw a lot of activity around 2.6.30 to get the dom0 patches accepted
into the mainline kernel, but now while I believe the 2.6.31 merge
window is open I see very little activity. There was an interesting
thread prior to the merge window opening on the subject of dom0 support
and wondered if as a result of this there's been a decision not to try
to get the patches accepted at this time? If this is the case is there a
plan to try again in the future?

I also saw a thread from Kier recently on the subject of the xenbits
linux trees, and wondered if any decision came out of that? Within the
thread I saw mention of a number of other kernel patch sets that offer
dom0 in a more recent kernels, but certain features seem to be missing
from each one.

If I wanted to to use a more recent kernel for dom0 within the next 3
months what options are available, and do these options have any
limitations?

thanks,

Anthony Wright.

Re: What is the current state of Dom0 kernel support?

[Thiago Camargo Martins Cordeiro]

[-- Attachment #1.1: Type: text/plain, Size: 1680 bytes --]

Well, I'm using the Linux 2.6.26 as dom0 since december 2008, from Debian
Lenny, it is pretty stable and supports modern hardware without problems...
Both 32 and 64 bits are working very well in my datacenter, I have +50 Xens
based on Debian... Also, OpenSUSE 11 have Linux 2.6.27 with dom0 support...

Cheers!
Thiago

2009/6/18 Anthony Wright <anthony@overnetdata.com>

> I've seen a number of posts relating to dom0 support in more recent
> kernel versions, but I'm unsure what the current situation is and
> wondered if somebody might be able clarify things.
>
> I saw a lot of activity around 2.6.30 to get the dom0 patches accepted
> into the mainline kernel, but now while I believe the 2.6.31 merge
> window is open I see very little activity. There was an interesting
> thread prior to the merge window opening on the subject of dom0 support
> and wondered if as a result of this there's been a decision not to try
> to get the patches accepted at this time? If this is the case is there a
> plan to try again in the future?
>
> I also saw a thread from Kier recently on the subject of the xenbits
> linux trees, and wondered if any decision came out of that? Within the
> thread I saw mention of a number of other kernel patch sets that offer
> dom0 in a more recent kernels, but certain features seem to be missing
> from each one.
>
> If I wanted to to use a more recent kernel for dom0 within the next 3
> months what options are available, and do these options have any
> limitations?
>
> thanks,
>
> Anthony Wright.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 2182 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: What is the current state of Dom0 kernel support?

[Dennis Krul]

[-- Attachment #1.1: Type: text/plain, Size: 2212 bytes --]

I guess we' re in the same boat. These are my experiences so far:

Based on the discussions on LKML and xen-devel I believe it's highly
unlikely the code for dom0 support will make it in mainline any time soon,
if ever.

There are several git trees here and there, but none of them is both stable
and up-to-date (or even actively maintained).

Michael Young maintains dom0 packages for Fedora. Although I appreciate his
work, his kernels are not stable enough for production use. These packages
are not in the main repositories (http://fedorapeople.org/~myoung/dom0/)

OpenSUSE has a kernel-xen package in factory that's based on 2.6.30, but
development of this package is happening behind closed doors. I'd like to
see a more public effort to create a stable dom0 kernel instead of relying
on (and trusting) the work of others.

So in my opinion what needs to happen is create a new tracking branch for
mainline, apply all the patches that are not being accepted upstream, and
work from there to get something that's stable. It should not be too
difficult to keep tracking mainline and release dom0 kernels as new vanilla
kernels come out.

That's ofcourse easier said then done. There will be merge conflicts and
other issues. It will take quite some time to keep track of all patches,
test them, make packages, and so on. It's obviously not something you can do
by your own. But I have the feeling there are enough people on this list
able and willing to help out. If we concentrate our efforts I believe we can
make something happen.

Ideally, some core Xen hackers would support this effort, but even if they
don't (due to time constraints or for whatever reason), I believe we can
pull this off.

So I was kind of wondering: Do you think this might work? And who would be
willing to help out? You don't need to be a hardcore kernel hacker (I'm
not), but one or two people with in depth kernel knowledge might come in
handy sometimes :). Hosting will not be a problem (I can provide a dedicated
VPS, everyone who wants to help can get commit and/or shell access, whatever
we need to make this work).

We have 2 people (myself included) who can help out. Anyone else? :)

-- 
Dennis Krul <dweazle@gmail.com>

[-- Attachment #1.2: Type: text/html, Size: 2452 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: What is the current state of Dom0 kernel support?

[Jeremy Fitzhardinge]

On 06/18/09 15:35, Anthony Wright wrote:
> I've seen a number of posts relating to dom0 support in more recent
> kernel versions, but I'm unsure what the current situation is and
> wondered if somebody might be able clarify things.
>
> I saw a lot of activity around 2.6.30 to get the dom0 patches accepted
> into the mainline kernel, but now while I believe the 2.6.31 merge
> window is open I see very little activity. There was an interesting
> thread prior to the merge window opening on the subject of dom0 support
> and wondered if as a result of this there's been a decision not to try
> to get the patches accepted at this time? If this is the case is there a
> plan to try again in the future?
>
> I also saw a thread from Kier recently on the subject of the xenbits
> linux trees, and wondered if any decision came out of that? Within the
> thread I saw mention of a number of other kernel patch sets that offer
> dom0 in a more recent kernels, but certain features seem to be missing
> from each one.
>
> If I wanted to to use a more recent kernel for dom0 within the next 3
> months what options are available, and do these options have any
> limitations?
>   

Well, the quick overview is:

    * I will be maintaining xen.git with dom0 support in it, and keeping
      that close to current kernel.org
    * A number of kernel developers expressed fairly strong objections
      to some aspects of the current dom0 patches, so I'm going over
      them to come up with something that will be more acceptable.  At
      the moment that looks like I'll be making changes to Xen itself,
      as well as the kernel, so there will likely be a requirement to
      track fairly up-to-date versions of xen-unstable for a while
    * I will be upstreaming pieces of dom0 support as they become ready
      and look acceptable for upstreaming.  The goal is to start
      shrinking the side of the out-of-tree patches as core kernel
      support increases.  This is going to be an incremental process,
      but it will work much better than trying to shove everything in at
      once.  The smaller (and less intrusive) the out-of-tree patchset
      becomes, the easier it is for distros to include it in their own
      patchset.

For now I've backed off putting anything much into the current merge
window, or even the next, in favour of getting the tree into good shape
for Xen users and working out what needs to happen to get a solid set of
upstreamable patches.

    J

Re: What is the current state of Dom0 kernel support?

[Tim Post]

Hi Jeremy,

On Fri, 2009-06-26 at 11:04 -0700, Jeremy Fitzhardinge wrote:

> For now I've backed off putting anything much into the current merge
> window, or even the next, in favour of getting the tree into good shape
> for Xen users and working out what needs to happen to get a solid set of
> upstreamable patches.

Is it possible for you to set up a blog just for this? I think many
people are just going to pull your tree, it would be really, really nice
to have a feed to pull so we know when to pull and update .. especially
when the next merge window closes.

As if you didn't have your hands full already :) Perhaps something on
xen.org just for kernel development?

Sorry if something like this already exists.

Cheers,
--Tim

Re: What is the current state of Dom0 kernel support?

[Jeremy Fitzhardinge]

On 06/26/09 11:21, Tim Post wrote:
> Is it possible for you to set up a blog just for this? I think many
> people are just going to pull your tree, it would be really, really nice
> to have a feed to pull so we know when to pull and update .. especially
> when the next merge window closes.
>
> As if you didn't have your hands full already :) Perhaps something on
> xen.org just for kernel development?
>
> Sorry if something like this already exists.
>   

No, its a good idea.  I'll sort something out (and poke me if I don't
appear to do anything in the next few days).

    J

Re: What is the current state of Dom0 kernel support?

[Pasi Kärkkäinen]

On Fri, Jun 26, 2009 at 11:29:30AM -0700, Jeremy Fitzhardinge wrote:
> On 06/26/09 11:21, Tim Post wrote:
> > Is it possible for you to set up a blog just for this? I think many
> > people are just going to pull your tree, it would be really, really nice
> > to have a feed to pull so we know when to pull and update .. especially
> > when the next merge window closes.
> >
> > As if you didn't have your hands full already :) Perhaps something on
> > xen.org just for kernel development?
> >
> > Sorry if something like this already exists.
> >   
> 
> No, its a good idea.  I'll sort something out (and poke me if I don't
> appear to do anything in the next few days).
> 

Btw what's the current tree people should be testing? xen-tip/next or
rebase/master? 

-- Pasi

Re: What is the current state of Dom0 kernel support?

[Jeremy Fitzhardinge]

On 07/08/09 15:14, Pasi Kärkkäinen wrote:
> On Fri, Jun 26, 2009 at 11:29:30AM -0700, Jeremy Fitzhardinge wrote:
>   
>> On 06/26/09 11:21, Tim Post wrote:
>>     
>>> Is it possible for you to set up a blog just for this? I think many
>>> people are just going to pull your tree, it would be really, really nice
>>> to have a feed to pull so we know when to pull and update .. especially
>>> when the next merge window closes.
>>>
>>> As if you didn't have your hands full already :) Perhaps something on
>>> xen.org just for kernel development?
>>>
>>> Sorry if something like this already exists.
>>>   
>>>       
>> No, its a good idea.  I'll sort something out (and poke me if I don't
>> appear to do anything in the next few days).
>>
>>     
>
> Btw what's the current tree people should be testing? xen-tip/next or
> rebase/master? 

rebase/master is what I'm currently working on.  It's work-in-progress,
but it works for me at the moment.  I'd appreciate any test results you
have.  (I don't yet have a fix in there for your PAE issue however.)

I'm planning on renaming these branches to xen/... and proposing they
become the basis for ongoing work.

    J

Re: What is the current state of Dom0 kernel support?

[M A Young]

On Wed, 8 Jul 2009, Jeremy Fitzhardinge wrote:

> rebase/master is what I'm currently working on.  It's work-in-progress,
> but it works for me at the moment.  I'd appreciate any test results you
> have.  (I don't yet have a fix in there for your PAE issue however.)

x86_64 works for me. i686 does the following then stops. This is from 
rebase/master from 30th June.

 	Michael Young

  __  __            _____  _____  _    _ _    __      _ _
  \ \/ /___ _ __   |___ / |___ / / |  / / |  / _| ___/ / |
   \  // _ \ '_ \    |_ \   |_ \ | |__| | | | |_ / __| | |
   /  \  __/ | | |  ___) | ___) || |__| | |_|  _| (__| | |
  /_/\_\___|_| |_| |____(_)____(_)_|  |_|_(_)_|  \___|_|_|

(XEN) Xen version 3.3.1-11.fc11 (mockbuild@(none)) (gcc version 4.4.0 
20090307 (Red Hat 4.4.0-0.23) (GCC) ) Tue Mar 10 08:26:32 EDT 2009
(XEN) Latest ChangeSet: unavailable
(XEN) Command line: com1=38400 console=com1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009fc00 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000001f740000 (usable)
(XEN)  000000001f740000 - 000000001f750000 (ACPI data)
(XEN)  000000001f750000 - 000000001f800000 (ACPI NVS)
(XEN) System RAM: 502MB (514940kB)
(XEN) ACPI: RSDP 000F7970, 0014 (r0 ACPIAM)
(XEN) ACPI: RSDT 1F740000, 0030 (r1 INTEL  D845GRG  20020909 MSFT 
97)
(XEN) ACPI: FACP 1F740200, 0081 (r2 INTEL  D845GRG  20020909 MSFT 
97)
(XEN) ACPI: DSDT 1F740400, 3F21 (r1 INTEL  D845GRG       10A MSFT 
100000D)
(XEN) ACPI: FACS 1F750000, 0040
(XEN) ACPI: APIC 1F740300, 0068 (r1 INTEL  D845GRG  20020909 MSFT 
97)
(XEN) ACPI: ASF! 1F744330, 0084 (r16 AMIASF I845GASF        1 MSFT 
100000D)
(XEN) Xen heap: 9MB (9732kB)
(XEN) Domain heap initialised
(XEN) Processor #0 15:2 APIC version 20
(XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2400.128 MHz processor.
(XEN) CPU0: Intel(R) Pentium(R) 4 CPU 2.40GHz stepping 07
(XEN) Total of 1 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer is 3.579MHz ACPI PM Timer
(XEN) Brought up 1 CPUs
(XEN) I/O virtualisation disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 32-bit, PAE, lsb
(XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0x400000 -> 0x134f000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000001a000000->000000001c000000 (108521 pages to 
be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: c0400000->c134f000
(XEN)  Init. ramdisk: c134f000->c1ab0000
(XEN)  Phys-Mach map: c1ab0000->c1b21fa4
(XEN)  Start info:    c1b22000->c1b22474
(XEN)  Page tables:   c1b23000->c1b36000
(XEN)  Boot stack:    c1b36000->c1b37000
(XEN)  TOTAL:         c0000000->c1c00000
(XEN)  ENTRY ADDRESS: c09e6000
(XEN) Dom0 has maximum 1 VCPUs
(XEN) Scrubbing Free RAM: done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input 
to Xen)
(XEN) Freed 108kB init memory.
mapping kernel into physical memory
Xen: setup ISA identity maps
about to get started...
(XEN) ioapic_guest_write: apic=0, pin=2, old_irq=0, new_irq=-1
(XEN) ioapic_guest_write: old_entry=000009f0, new_entry=00010900
(XEN) ioapic_guest_write: Attempt to remove IO-APIC pin of in-use IRQ!
(XEN) ioapic_guest_write: apic=0, pin=4, old_irq=4, new_irq=-1
(XEN) ioapic_guest_write: old_entry=000009f1, new_entry=00010900
(XEN) ioapic_guest_write: Attempt to remove IO-APIC pin of in-use IRQ!

Re: Xen Source Code Overview Document

[Zhigang Wang]

A wiki page for it should be better. see: http://wiki.xensource.com/xenwiki/XenSourceCode

thanks,

zhigang

Stephen Spector wrote:
> Xen Developers:
> 
>  
> 
> The response from the community survey indicated that people were
> looking for a Xen source code overview document that would help them
> find particular files based on the feature they were interested in. To
> support this, I have created a document (see attached in .odt and .doc)
> that lists every file in the xen source tree as well as a brief
> description of the content in each file; where I could understand the
> code).  If you know what is in a file that has no text next to it;
> please let me know and I will update the documents.
> 
>  
> 
> The next step of this process is to write a brief description of each
> directory to help people find the code they need to examine.  I need
> some help here.  If just a few developers took 10 minutes a day to write
> a brief description of 1 or 2 directories a day I could have this
> document completed by the end of the week.
> 
>  
> 
> Thanks for your help in finishing this document.
> 
>  
> 
> PS – I have also posted a new Configuration document that lists all the
> possible Domain configuration file settings; this document is in review
> on xen-users.
> 
>  
> 
> Stephen Spector
> 
> Community Manager, Xen.org
> 
> stephen.spector@xen.org <mailto:stephen.spector@xen.org>
> 
> (772) 621-5062
> 
> BLOG: http://blog.xen.org <http://blog.xen.org/>
> 
> Folllow Me on Twitter: http://twitter.com/xen_com_mgr
> 
>  
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: What is the current state of Dom0 kernel support?

[Pasi Kärkkäinen]

On Wed, Jul 08, 2009 at 05:23:36PM -0700, Jeremy Fitzhardinge wrote:
> On 07/08/09 15:14, Pasi Kärkkäinen wrote:
> > On Fri, Jun 26, 2009 at 11:29:30AM -0700, Jeremy Fitzhardinge wrote:
> >   
> >> On 06/26/09 11:21, Tim Post wrote:
> >>     
> >>> Is it possible for you to set up a blog just for this? I think many
> >>> people are just going to pull your tree, it would be really, really nice
> >>> to have a feed to pull so we know when to pull and update .. especially
> >>> when the next merge window closes.
> >>>
> >>> As if you didn't have your hands full already :) Perhaps something on
> >>> xen.org just for kernel development?
> >>>
> >>> Sorry if something like this already exists.
> >>>   
> >>>       
> >> No, its a good idea.  I'll sort something out (and poke me if I don't
> >> appear to do anything in the next few days).
> >>
> >>     
> >
> > Btw what's the current tree people should be testing? xen-tip/next or
> > rebase/master? 
> 
> rebase/master is what I'm currently working on.  It's work-in-progress,
> but it works for me at the moment.  I'd appreciate any test results you
> have.  (I don't yet have a fix in there for your PAE issue however.)
> 

I just tried latest rebase/master (2.6.31-rc1) on Xen 3.4.0.

Seems to crash during dom0 kernel startup:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-09-rebase-master-with-highpte.txt

Checking if this processor honours the WP bit even in supervisor mode...
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<c058d0e7>] xen_evtchn_do_upcall+0xcc/0x13f
*pdpt = 000000003d272001 
Thread overran stack, or stack corrupted
Oops: 0000 [#1] SMP 

It's 32bit PAE like usual..

My previous xen-tip/next 2.6.30-rc6-tip booted ok.

-- Pasi

Re: What is the current state of Dom0 kernel support?

[Pasi Kärkkäinen]

On Fri, Jul 10, 2009 at 01:24:14AM +0300, Pasi Kärkkäinen wrote:
> On Wed, Jul 08, 2009 at 05:23:36PM -0700, Jeremy Fitzhardinge wrote:
> > On 07/08/09 15:14, Pasi Kärkkäinen wrote:
> > > On Fri, Jun 26, 2009 at 11:29:30AM -0700, Jeremy Fitzhardinge wrote:
> > >   
> > >> On 06/26/09 11:21, Tim Post wrote:
> > >>     
> > >>> Is it possible for you to set up a blog just for this? I think many
> > >>> people are just going to pull your tree, it would be really, really nice
> > >>> to have a feed to pull so we know when to pull and update .. especially
> > >>> when the next merge window closes.
> > >>>
> > >>> As if you didn't have your hands full already :) Perhaps something on
> > >>> xen.org just for kernel development?
> > >>>
> > >>> Sorry if something like this already exists.
> > >>>   
> > >>>       
> > >> No, its a good idea.  I'll sort something out (and poke me if I don't
> > >> appear to do anything in the next few days).
> > >>
> > >>     
> > >
> > > Btw what's the current tree people should be testing? xen-tip/next or
> > > rebase/master? 
> > 
> > rebase/master is what I'm currently working on.  It's work-in-progress,
> > but it works for me at the moment.  I'd appreciate any test results you
> > have.  (I don't yet have a fix in there for your PAE issue however.)
> > 
> 
> I just tried latest rebase/master (2.6.31-rc1) on Xen 3.4.0.
> 
> Seems to crash during dom0 kernel startup:
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-09-rebase-master-with-highpte.txt
> 
> Checking if this processor honours the WP bit even in supervisor mode...
> BUG: unable to handle kernel NULL pointer dereference at (null)
> IP: [<c058d0e7>] xen_evtchn_do_upcall+0xcc/0x13f
> *pdpt = 000000003d272001 
> Thread overran stack, or stack corrupted
> Oops: 0000 [#1] SMP 
> 
> It's 32bit PAE like usual..
> 
> My previous xen-tip/next 2.6.30-rc6-tip booted ok.
> 

Is there something to test to debug this further? 

-- Pasi

Re: What is the current state of Dom0 kernel support? / crash

[Pasi Kärkkäinen]

On Wed, Jul 15, 2009 at 11:22:42AM +0300, Pasi Kärkkäinen wrote:
> On Fri, Jul 10, 2009 at 01:24:14AM +0300, Pasi Kärkkäinen wrote:
> > On Wed, Jul 08, 2009 at 05:23:36PM -0700, Jeremy Fitzhardinge wrote:
> > > On 07/08/09 15:14, Pasi Kärkkäinen wrote:
> > > > On Fri, Jun 26, 2009 at 11:29:30AM -0700, Jeremy Fitzhardinge wrote:
> > > >   
> > > >> On 06/26/09 11:21, Tim Post wrote:
> > > >>     
> > > >>> Is it possible for you to set up a blog just for this? I think many
> > > >>> people are just going to pull your tree, it would be really, really nice
> > > >>> to have a feed to pull so we know when to pull and update .. especially
> > > >>> when the next merge window closes.
> > > >>>
> > > >>> As if you didn't have your hands full already :) Perhaps something on
> > > >>> xen.org just for kernel development?
> > > >>>
> > > >>> Sorry if something like this already exists.
> > > >>>   
> > > >>>       
> > > >> No, its a good idea.  I'll sort something out (and poke me if I don't
> > > >> appear to do anything in the next few days).
> > > >>
> > > >>     
> > > >
> > > > Btw what's the current tree people should be testing? xen-tip/next or
> > > > rebase/master? 
> > > 
> > > rebase/master is what I'm currently working on.  It's work-in-progress,
> > > but it works for me at the moment.  I'd appreciate any test results you
> > > have.  (I don't yet have a fix in there for your PAE issue however.)
> > > 
> > 
> > I just tried latest rebase/master (2.6.31-rc1) on Xen 3.4.0.
> > 
> > Seems to crash during dom0 kernel startup:
> > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-09-rebase-master-with-highpte.txt
> > 
> > Checking if this processor honours the WP bit even in supervisor mode...
> > BUG: unable to handle kernel NULL pointer dereference at (null)
> > IP: [<c058d0e7>] xen_evtchn_do_upcall+0xcc/0x13f
> > *pdpt = 000000003d272001 
> > Thread overran stack, or stack corrupted
> > Oops: 0000 [#1] SMP 
> > 
> > It's 32bit PAE like usual..
> > 
> > My previous xen-tip/next 2.6.30-rc6-tip booted ok.
> > 
> 
> Is there something to test to debug this further? 
> 

I just tried the latest 32b PAE rebase/master tree (2.6.31-rc3).

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-10-rebase-master-with-highpte.txt

Checking if this processor honours the WP bit even in supervisor mode...
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f
*pdpt = 000000003d275001 
Thread overran stack, or stack corrupted
Oops: 0000 [#1] SMP 
last sysfs file: 
Modules linked in:

Pid: 0, comm: swapper Not tainted (2.6.31-rc3 #20) P8SC8
EIP: 0061:[<c058cdcb>] EFLAGS: 00010046 CPU: 0
EIP is at xen_evtchn_do_upcall+0xcc/0x13f
EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
ESI: 00000000 EDI: c08ec558 EBP: c087eedc ESP: c087eea0
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
Stack:
 00001a6e 00000220 00000200 00000000 00000000 00000000 e3201014 c08ec558
<0> c087eee4 f5681000 e3201010 00000000 00000000 c09017f8 f54ff000 c087ef20
<0> c0409927 00000000 c09017f8 f54ff000 c09017f8 f54ff000 c087ef20 c0843f70
Call Trace:
 [<c0409927>] ? xen_do_upcall+0x7/0xc
 [<c0404581>] ? xen_pte_clear+0x9/0x12
 [<c0427a94>] ? set_pte_vaddr+0xb4/0xc4
 [<c0426c8c>] ? __native_set_fixmap+0x25/0x30
 [<c040471a>] ? xen_set_fixmap+0xc7/0xcc
 [<c0897d86>] ? mem_init+0x24a/0x298
 [<c088367e>] ? start_kernel+0x14b/0x2cd
 [<c088336f>] ? unknown_bootoption+0x0/0x18e
 [<c0883082>] ? i386_start_kernel+0x71/0x79
 [<c0886188>] ? xen_start_kernel+0x52a/0x533
Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
8b 55 e4 e8 36 de e7 ff 8b 55 f0 8b 45 d0 03 
05 1c 0c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
18 
EIP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
CR2: 0000000000000000
---[ end trace 4eaa2a86a8e2da22 ]---
Kernel panic - not syncing: Fatal exception in interrupt


-- Pasi

Re: What is the current state of Dom0 kernel support? / crash

[Jeremy Fitzhardinge]

On 07/21/09 06:03, Pasi Kärkkäinen wrote:
> I just tried the latest 32b PAE rebase/master tree (2.6.31-rc3).
>
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-10-rebase-master-with-highpte.txt
>
> Checking if this processor honours the WP bit even in supervisor mode...
> BUG: unable to handle kernel NULL pointer dereference at (null)
> IP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f
> *pdpt = 000000003d275001 
> Thread overran stack, or stack corrupted
> Oops: 0000 [#1] SMP 
> last sysfs file: 
> Modules linked in:
>
> Pid: 0, comm: swapper Not tainted (2.6.31-rc3 #20) P8SC8
> EIP: 0061:[<c058cdcb>] EFLAGS: 00010046 CPU: 0
> EIP is at xen_evtchn_do_upcall+0xcc/0x13f
> EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
> ESI: 00000000 EDI: c08ec558 EBP: c087eedc ESP: c087eea0
>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
> Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
> Stack:
>  00001a6e 00000220 00000200 00000000 00000000 00000000 e3201014 c08ec558
> <0> c087eee4 f5681000 e3201010 00000000 00000000 c09017f8 f54ff000 c087ef20
> <0> c0409927 00000000 c09017f8 f54ff000 c09017f8 f54ff000 c087ef20 c0843f70
> Call Trace:
>  [<c0409927>] ? xen_do_upcall+0x7/0xc
>  [<c0404581>] ? xen_pte_clear+0x9/0x12
>  [<c0427a94>] ? set_pte_vaddr+0xb4/0xc4
>  [<c0426c8c>] ? __native_set_fixmap+0x25/0x30
>  [<c040471a>] ? xen_set_fixmap+0xc7/0xcc
>  [<c0897d86>] ? mem_init+0x24a/0x298
>  [<c088367e>] ? start_kernel+0x14b/0x2cd
>  [<c088336f>] ? unknown_bootoption+0x0/0x18e
>  [<c0883082>] ? i386_start_kernel+0x71/0x79
>  [<c0886188>] ? xen_start_kernel+0x52a/0x533
> Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
> 8b 55 e4 e8 36 de e7 ff 8b 55 f0 8b 45 d0 03 
> 05 1c 0c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
> 18 
> EIP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
> CR2: 0000000000000000
> ---[ end trace 4eaa2a86a8e2da22 ]---
> Kernel panic - not syncing: Fatal exception in interrupt
>   

Haven't seen that one before.  The stack backtrace is a bit fuzzy; do
you have CONFIG_FRAMEPOINTER enabled?  And if you have CONFIG_DEBUGINFO
enabled, you can map the eip c058cdcb to a specific source line (its not
clear to me which pointer is NULL).

Thanks,
    J

Re: What is the current state of Dom0 kernel support? / crash

[Pasi Kärkkäinen]

On Wed, Jul 22, 2009 at 12:14:37PM -0700, Jeremy Fitzhardinge wrote:
> On 07/21/09 06:03, Pasi Kärkkäinen wrote:
> > I just tried the latest 32b PAE rebase/master tree (2.6.31-rc3).
> >
> > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-10-rebase-master-with-highpte.txt
> >
> > Checking if this processor honours the WP bit even in supervisor mode...
> > BUG: unable to handle kernel NULL pointer dereference at (null)
> > IP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f
> > *pdpt = 000000003d275001 
> > Thread overran stack, or stack corrupted
> > Oops: 0000 [#1] SMP 
> > last sysfs file: 
> > Modules linked in:
> >
> > Pid: 0, comm: swapper Not tainted (2.6.31-rc3 #20) P8SC8
> > EIP: 0061:[<c058cdcb>] EFLAGS: 00010046 CPU: 0
> > EIP is at xen_evtchn_do_upcall+0xcc/0x13f
> > EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
> > ESI: 00000000 EDI: c08ec558 EBP: c087eedc ESP: c087eea0
> >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
> > Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
> > Stack:
> >  00001a6e 00000220 00000200 00000000 00000000 00000000 e3201014 c08ec558
> > <0> c087eee4 f5681000 e3201010 00000000 00000000 c09017f8 f54ff000 c087ef20
> > <0> c0409927 00000000 c09017f8 f54ff000 c09017f8 f54ff000 c087ef20 c0843f70
> > Call Trace:
> >  [<c0409927>] ? xen_do_upcall+0x7/0xc
> >  [<c0404581>] ? xen_pte_clear+0x9/0x12
> >  [<c0427a94>] ? set_pte_vaddr+0xb4/0xc4
> >  [<c0426c8c>] ? __native_set_fixmap+0x25/0x30
> >  [<c040471a>] ? xen_set_fixmap+0xc7/0xcc
> >  [<c0897d86>] ? mem_init+0x24a/0x298
> >  [<c088367e>] ? start_kernel+0x14b/0x2cd
> >  [<c088336f>] ? unknown_bootoption+0x0/0x18e
> >  [<c0883082>] ? i386_start_kernel+0x71/0x79
> >  [<c0886188>] ? xen_start_kernel+0x52a/0x533
> > Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
> > 8b 55 e4 e8 36 de e7 ff 8b 55 f0 8b 45 d0 03 
> > 05 1c 0c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
> > 18 
> > EIP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
> > CR2: 0000000000000000
> > ---[ end trace 4eaa2a86a8e2da22 ]---
> > Kernel panic - not syncing: Fatal exception in interrupt
> >   
> 
> Haven't seen that one before.  
>

Ok. I've seen many people report crashes during startup with rebase/master
on 32b PAE. I assume they're seeing this same issue.

> The stack backtrace is a bit fuzzy; do you have CONFIG_FRAMEPOINTER enabled?  
> And if you have CONFIG_DEBUGINFO enabled, you can map the eip c058cdcb 
> to a specific source line (its not clear to me which pointer is NULL).
> 

[root@dom0test linux-2.6-xen]# grep -i CONFIG_FRAMEPOINTER .config
[root@dom0test linux-2.6-xen]# grep -i CONFIG_DEBUGINFO .config
[root@dom0test linux-2.6-xen]# 

Unfortunately those were not enabled.. I'll build a new kernel with
CONFIG_DEBUGINFO enabled.

-- Pasi

Re: What is the current state of Dom0 kernel support? / crash

[Jeremy Fitzhardinge]

On 07/22/09 12:35, Pasi Kärkkäinen wrote:
> On Wed, Jul 22, 2009 at 12:14:37PM -0700, Jeremy Fitzhardinge wrote:
>   
>> On 07/21/09 06:03, Pasi Kärkkäinen wrote:
>>     
>>> I just tried the latest 32b PAE rebase/master tree (2.6.31-rc3).
>>>
>>> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-10-rebase-master-with-highpte.txt
>>>
>>> Checking if this processor honours the WP bit even in supervisor mode...
>>> BUG: unable to handle kernel NULL pointer dereference at (null)
>>> IP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f
>>> *pdpt = 000000003d275001 
>>> Thread overran stack, or stack corrupted
>>> Oops: 0000 [#1] SMP 
>>> last sysfs file: 
>>> Modules linked in:
>>>
>>> Pid: 0, comm: swapper Not tainted (2.6.31-rc3 #20) P8SC8
>>> EIP: 0061:[<c058cdcb>] EFLAGS: 00010046 CPU: 0
>>> EIP is at xen_evtchn_do_upcall+0xcc/0x13f
>>> EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
>>> ESI: 00000000 EDI: c08ec558 EBP: c087eedc ESP: c087eea0
>>>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
>>> Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
>>> Stack:
>>>  00001a6e 00000220 00000200 00000000 00000000 00000000 e3201014 c08ec558
>>> <0> c087eee4 f5681000 e3201010 00000000 00000000 c09017f8 f54ff000 c087ef20
>>> <0> c0409927 00000000 c09017f8 f54ff000 c09017f8 f54ff000 c087ef20 c0843f70
>>> Call Trace:
>>>  [<c0409927>] ? xen_do_upcall+0x7/0xc
>>>  [<c0404581>] ? xen_pte_clear+0x9/0x12
>>>  [<c0427a94>] ? set_pte_vaddr+0xb4/0xc4
>>>  [<c0426c8c>] ? __native_set_fixmap+0x25/0x30
>>>  [<c040471a>] ? xen_set_fixmap+0xc7/0xcc
>>>  [<c0897d86>] ? mem_init+0x24a/0x298
>>>  [<c088367e>] ? start_kernel+0x14b/0x2cd
>>>  [<c088336f>] ? unknown_bootoption+0x0/0x18e
>>>  [<c0883082>] ? i386_start_kernel+0x71/0x79
>>>  [<c0886188>] ? xen_start_kernel+0x52a/0x533
>>> Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
>>> 8b 55 e4 e8 36 de e7 ff 8b 55 f0 8b 45 d0 03 
>>> 05 1c 0c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
>>> 18 
>>> EIP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
>>> CR2: 0000000000000000
>>> ---[ end trace 4eaa2a86a8e2da22 ]---
>>> Kernel panic - not syncing: Fatal exception in interrupt
>>>   
>>>       
>> Haven't seen that one before.  
>>
>>     
>
> Ok. I've seen many people report crashes during startup with rebase/master
> on 32b PAE. I assume they're seeing this same issue.
>   

Unfortunately its a bit awkward for me to test 32-bit regularly, so I
rely on other reports.  IanC did send me a success report from the -rc1
kernel, but its quite likely something changed since then.  It shouldn't
be too hard to work out once we can pinpoint the crash site.

>> The stack backtrace is a bit fuzzy; do you have CONFIG_FRAMEPOINTER enabled?  
>> And if you have CONFIG_DEBUGINFO enabled, you can map the eip c058cdcb 
>> to a specific source line (its not clear to me which pointer is NULL).
>>
>>     
>
> [root@dom0test linux-2.6-xen]# grep -i CONFIG_FRAMEPOINTER .config
> [root@dom0test linux-2.6-xen]# grep -i CONFIG_DEBUGINFO .config
> [root@dom0test linux-2.6-xen]# 
>
> Unfortunately those were not enabled.. I'll build a new kernel with
> CONFIG_DEBUGINFO enabled.
>   

Thanks,
    J

Re: What is the current state of Dom0 kernel support? / crash

[Pasi Kärkkäinen]

On Wed, Jul 22, 2009 at 10:35:30PM +0300, Pasi Kärkkäinen wrote:
> On Wed, Jul 22, 2009 at 12:14:37PM -0700, Jeremy Fitzhardinge wrote:
> > On 07/21/09 06:03, Pasi Kärkkäinen wrote:
> > > I just tried the latest 32b PAE rebase/master tree (2.6.31-rc3).
> > >
> > > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-10-rebase-master-with-highpte.txt
> > >
> > > Checking if this processor honours the WP bit even in supervisor mode...
> > > BUG: unable to handle kernel NULL pointer dereference at (null)
> > > IP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f
> > > *pdpt = 000000003d275001 
> > > Thread overran stack, or stack corrupted
> > > Oops: 0000 [#1] SMP 
> > > last sysfs file: 
> > > Modules linked in:
> > >
> > > Pid: 0, comm: swapper Not tainted (2.6.31-rc3 #20) P8SC8
> > > EIP: 0061:[<c058cdcb>] EFLAGS: 00010046 CPU: 0
> > > EIP is at xen_evtchn_do_upcall+0xcc/0x13f
> > > EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
> > > ESI: 00000000 EDI: c08ec558 EBP: c087eedc ESP: c087eea0
> > >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
> > > Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
> > > Stack:
> > >  00001a6e 00000220 00000200 00000000 00000000 00000000 e3201014 c08ec558
> > > <0> c087eee4 f5681000 e3201010 00000000 00000000 c09017f8 f54ff000 c087ef20
> > > <0> c0409927 00000000 c09017f8 f54ff000 c09017f8 f54ff000 c087ef20 c0843f70
> > > Call Trace:
> > >  [<c0409927>] ? xen_do_upcall+0x7/0xc
> > >  [<c0404581>] ? xen_pte_clear+0x9/0x12
> > >  [<c0427a94>] ? set_pte_vaddr+0xb4/0xc4
> > >  [<c0426c8c>] ? __native_set_fixmap+0x25/0x30
> > >  [<c040471a>] ? xen_set_fixmap+0xc7/0xcc
> > >  [<c0897d86>] ? mem_init+0x24a/0x298
> > >  [<c088367e>] ? start_kernel+0x14b/0x2cd
> > >  [<c088336f>] ? unknown_bootoption+0x0/0x18e
> > >  [<c0883082>] ? i386_start_kernel+0x71/0x79
> > >  [<c0886188>] ? xen_start_kernel+0x52a/0x533
> > > Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
> > > 8b 55 e4 e8 36 de e7 ff 8b 55 f0 8b 45 d0 03 
> > > 05 1c 0c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
> > > 18 
> > > EIP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
> > > CR2: 0000000000000000
> > > ---[ end trace 4eaa2a86a8e2da22 ]---
> > > Kernel panic - not syncing: Fatal exception in interrupt
> > >   
> > 
> > Haven't seen that one before.  
> >
> 
> Ok. I've seen many people report crashes during startup with rebase/master
> on 32b PAE. I assume they're seeing this same issue.
> 
> > The stack backtrace is a bit fuzzy; do you have CONFIG_FRAMEPOINTER enabled?  
> > And if you have CONFIG_DEBUGINFO enabled, you can map the eip c058cdcb 
> > to a specific source line (its not clear to me which pointer is NULL).
> > 
> 
> [root@dom0test linux-2.6-xen]# grep -i CONFIG_FRAMEPOINTER .config
> [root@dom0test linux-2.6-xen]# grep -i CONFIG_DEBUGINFO .config
> [root@dom0test linux-2.6-xen]# 
> 
> Unfortunately those were not enabled.. I'll build a new kernel with
> CONFIG_DEBUGINFO enabled.
> 

Actually CONFIG_DEBUG_INFO was enabled, if you meant that? 

(gdb) x/i 0xc058cdcb
0xc058cdcb <active_evtchns+124>:        mov    (%eax,%edx,1),%ecx

(gdb) disas 0xc058cdcb
Dump of assembler code for function active_evtchns:
0xc058cd4f <cpu_evtchn_mask+0>: shll   $0x7,-0x10(%ebp)
0xc058cd53 <xen_evtchn_do_upcall+84>:   mov    %edi,-0x20(%ebp)
0xc058cd56 <__xchg+10>: add    $0x4,%edx
0xc058cd59 <__xchg+13>: mov    %edx,-0x24(%ebp)
0xc058cd5c <xen_evtchn_do_upcall+93>:   mov    -0x14(%ebp),%ecx
0xc058cd5f <xen_evtchn_do_upcall+96>:   movb   $0x0,(%ecx)
0xc058cd62 <xen_evtchn_do_upcall+99>:   mov    %fs:0xc08ea60c,%eax
0xc058cd68 <xen_evtchn_do_upcall+105>:  add    %edi,%eax
0xc058cd6a <xen_evtchn_do_upcall+107>:  mov    (%eax),%ebx
0xc058cd6c <xen_evtchn_do_upcall+109>:  lea    0x1(%ebx),%edx
0xc058cd6f <xen_evtchn_do_upcall+112>:  test   %ebx,%ebx
0xc058cd71 <xen_evtchn_do_upcall+114>:  mov    %edx,(%eax)
0xc058cd73 <xen_evtchn_do_upcall+116>:  jne    0xc058ce28 <xen_evtchn_do_upcall+297>
0xc058cd79 <__xchg+45>: mov    -0x24(%ebp),%eax
0xc058cd7c <__xchg+48>: xchg   %ebx,(%eax)
0xc058cd7e <xen_evtchn_do_upcall+127>:  jmp    0xc058cdfb <xen_evtchn_do_upcall+252>
0xc058cd80 <__ffs+0>:   bsf    %ebx,%esi
0xc058cd83 <xen_evtchn_do_upcall+132>:  mov    %esi,%edx
0xc058cd85 <xen_evtchn_do_upcall+134>:  shl    $0x5,%edx
0xc058cd88 <xen_evtchn_do_upcall+137>:  mov    %edx,-0x2c(%ebp)
0xc058cd8b <active_evtchns+60>: lea    0x0(,%esi,4),%ecx
0xc058cd92 <active_evtchns+67>: lea    0x200(%esi),%eax
0xc058cd98 <active_evtchns+73>: lea    0x220(%esi),%edx
0xc058cd9e <active_evtchns+79>: mov    %ecx,-0x30(%ebp)
0xc058cda1 <active_evtchns+82>: mov    %eax,-0x34(%ebp)
0xc058cda4 <active_evtchns+85>: mov    %edx,-0x38(%ebp)
0xc058cda7 <xen_evtchn_do_upcall+168>:  jmp    0xc058cdbf <active_evtchns+112>
0xc058cda9 <__ffs+0>:   bsf    %eax,%ecx
0xc058cdac <xen_evtchn_do_upcall+173>:  add    -0x2c(%ebp),%ecx
0xc058cdaf <xen_evtchn_do_upcall+176>:  mov    (%edx,%ecx,4),%eax
0xc058cdb2 <xen_evtchn_do_upcall+179>:  cmp    $0xffffffff,%eax
0xc058cdb5 <xen_evtchn_do_upcall+182>:  je     0xc058cdaf <xen_evtchn_do_upcall+176>
0xc058cdb7 <xen_evtchn_do_upcall+184>:  mov    -0x1c(%ebp),%edx
0xc058cdba <xen_evtchn_do_upcall+187>:  call   0xc040abf5 <handle_irq>
0xc058cdbf <active_evtchns+112>:        mov    -0x10(%ebp),%edx
0xc058cdc2 <active_evtchns+115>:        mov    -0x30(%ebp),%eax
0xc058cdc5 <active_evtchns+118>:        add    0xc0970c1c,%eax
0xc058cdcb <active_evtchns+124>:        mov    (%eax,%edx,1),%ecx
0xc058cdce <active_evtchns+127>:        mov    -0x18(%ebp),%edx
0xc058cdd1 <active_evtchns+130>:        mov    -0x34(%ebp),%eax
0xc058cdd4 <active_evtchns+133>:        and    (%edx,%eax,4),%ecx
0xc058cdd7 <active_evtchns+136>:        mov    -0x38(%ebp),%eax
0xc058cdda <active_evtchns+139>:        mov    (%edx,%eax,4),%eax
0xc058cddd <xen_evtchn_do_upcall+222>:  mov    0xc0970c18,%edx
0xc058cde3 <active_evtchns+148>:        not    %eax
0xc058cde5 <active_evtchns+150>:        mov    %eax,-0x3c(%ebp)
End of assembler dump.
(gdb) 

Hopefully that helps..

-- Pasi

Re: What is the current state of Dom0 kernel support? / crash

[Pasi Kärkkäinen]

On Wed, Jul 22, 2009 at 12:55:11PM -0700, Jeremy Fitzhardinge wrote:
> On 07/22/09 12:35, Pasi Kärkkäinen wrote:
> > On Wed, Jul 22, 2009 at 12:14:37PM -0700, Jeremy Fitzhardinge wrote:
> >   
> >> On 07/21/09 06:03, Pasi Kärkkäinen wrote:
> >>     
> >>> I just tried the latest 32b PAE rebase/master tree (2.6.31-rc3).
> >>>
> >>> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-10-rebase-master-with-highpte.txt
> >>>
> >>> Checking if this processor honours the WP bit even in supervisor mode...
> >>> BUG: unable to handle kernel NULL pointer dereference at (null)
> >>> IP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f
> >>> *pdpt = 000000003d275001 
> >>> Thread overran stack, or stack corrupted
> >>> Oops: 0000 [#1] SMP 
> >>> last sysfs file: 
> >>> Modules linked in:
> >>>
> >>> Pid: 0, comm: swapper Not tainted (2.6.31-rc3 #20) P8SC8
> >>> EIP: 0061:[<c058cdcb>] EFLAGS: 00010046 CPU: 0
> >>> EIP is at xen_evtchn_do_upcall+0xcc/0x13f
> >>> EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
> >>> ESI: 00000000 EDI: c08ec558 EBP: c087eedc ESP: c087eea0
> >>>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
> >>> Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
> >>> Stack:
> >>>  00001a6e 00000220 00000200 00000000 00000000 00000000 e3201014 c08ec558
> >>> <0> c087eee4 f5681000 e3201010 00000000 00000000 c09017f8 f54ff000 c087ef20
> >>> <0> c0409927 00000000 c09017f8 f54ff000 c09017f8 f54ff000 c087ef20 c0843f70
> >>> Call Trace:
> >>>  [<c0409927>] ? xen_do_upcall+0x7/0xc
> >>>  [<c0404581>] ? xen_pte_clear+0x9/0x12
> >>>  [<c0427a94>] ? set_pte_vaddr+0xb4/0xc4
> >>>  [<c0426c8c>] ? __native_set_fixmap+0x25/0x30
> >>>  [<c040471a>] ? xen_set_fixmap+0xc7/0xcc
> >>>  [<c0897d86>] ? mem_init+0x24a/0x298
> >>>  [<c088367e>] ? start_kernel+0x14b/0x2cd
> >>>  [<c088336f>] ? unknown_bootoption+0x0/0x18e
> >>>  [<c0883082>] ? i386_start_kernel+0x71/0x79
> >>>  [<c0886188>] ? xen_start_kernel+0x52a/0x533
> >>> Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
> >>> 8b 55 e4 e8 36 de e7 ff 8b 55 f0 8b 45 d0 03 
> >>> 05 1c 0c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
> >>> 18 
> >>> EIP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
> >>> CR2: 0000000000000000
> >>> ---[ end trace 4eaa2a86a8e2da22 ]---
> >>> Kernel panic - not syncing: Fatal exception in interrupt
> >>>   
> >>>       
> >> Haven't seen that one before.  
> >>
> >>     
> >
> > Ok. I've seen many people report crashes during startup with rebase/master
> > on 32b PAE. I assume they're seeing this same issue.
> >   
> 
> Unfortunately its a bit awkward for me to test 32-bit regularly, so I
> rely on other reports.  IanC did send me a success report from the -rc1
> kernel, but its quite likely something changed since then.  It shouldn't
> be too hard to work out once we can pinpoint the crash site.
> 

I think I saw the same crash also on 2.6.31-rc1 .. 

-- Pasi

Re: What is the current state of Dom0 kernel support? / crash

[Jeremy Fitzhardinge]

On 07/22/09 12:57, Pasi Kärkkäinen wrote:
> On Wed, Jul 22, 2009 at 10:35:30PM +0300, Pasi Kärkkäinen wrote:
>   
>> On Wed, Jul 22, 2009 at 12:14:37PM -0700, Jeremy Fitzhardinge wrote:
>>     
>>> On 07/21/09 06:03, Pasi Kärkkäinen wrote:
>>>       
>>>> I just tried the latest 32b PAE rebase/master tree (2.6.31-rc3).
>>>>
>>>> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-10-rebase-master-with-highpte.txt
>>>>
>>>> Checking if this processor honours the WP bit even in supervisor mode...
>>>> BUG: unable to handle kernel NULL pointer dereference at (null)
>>>> IP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f
>>>> *pdpt = 000000003d275001 
>>>> Thread overran stack, or stack corrupted
>>>> Oops: 0000 [#1] SMP 
>>>> last sysfs file: 
>>>> Modules linked in:
>>>>
>>>> Pid: 0, comm: swapper Not tainted (2.6.31-rc3 #20) P8SC8
>>>> EIP: 0061:[<c058cdcb>] EFLAGS: 00010046 CPU: 0
>>>> EIP is at xen_evtchn_do_upcall+0xcc/0x13f
>>>> EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
>>>> ESI: 00000000 EDI: c08ec558 EBP: c087eedc ESP: c087eea0
>>>>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
>>>> Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
>>>> Stack:
>>>>  00001a6e 00000220 00000200 00000000 00000000 00000000 e3201014 c08ec558
>>>> <0> c087eee4 f5681000 e3201010 00000000 00000000 c09017f8 f54ff000 c087ef20
>>>> <0> c0409927 00000000 c09017f8 f54ff000 c09017f8 f54ff000 c087ef20 c0843f70
>>>> Call Trace:
>>>>  [<c0409927>] ? xen_do_upcall+0x7/0xc
>>>>  [<c0404581>] ? xen_pte_clear+0x9/0x12
>>>>  [<c0427a94>] ? set_pte_vaddr+0xb4/0xc4
>>>>  [<c0426c8c>] ? __native_set_fixmap+0x25/0x30
>>>>  [<c040471a>] ? xen_set_fixmap+0xc7/0xcc
>>>>  [<c0897d86>] ? mem_init+0x24a/0x298
>>>>  [<c088367e>] ? start_kernel+0x14b/0x2cd
>>>>  [<c088336f>] ? unknown_bootoption+0x0/0x18e
>>>>  [<c0883082>] ? i386_start_kernel+0x71/0x79
>>>>  [<c0886188>] ? xen_start_kernel+0x52a/0x533
>>>> Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
>>>> 8b 55 e4 e8 36 de e7 ff 8b 55 f0 8b 45 d0 03 
>>>> 05 1c 0c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
>>>> 18 
>>>> EIP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
>>>> CR2: 0000000000000000
>>>> ---[ end trace 4eaa2a86a8e2da22 ]---
>>>> Kernel panic - not syncing: Fatal exception in interrupt
>>>>   
>>>>         
>>> Haven't seen that one before.  
>>>
>>>       
>> Ok. I've seen many people report crashes during startup with rebase/master
>> on 32b PAE. I assume they're seeing this same issue.
>>
>>     
>>> The stack backtrace is a bit fuzzy; do you have CONFIG_FRAMEPOINTER enabled?  
>>> And if you have CONFIG_DEBUGINFO enabled, you can map the eip c058cdcb 
>>> to a specific source line (its not clear to me which pointer is NULL).
>>>
>>>       
>> [root@dom0test linux-2.6-xen]# grep -i CONFIG_FRAMEPOINTER .config
>> [root@dom0test linux-2.6-xen]# grep -i CONFIG_DEBUGINFO .config
>> [root@dom0test linux-2.6-xen]# 
>>
>> Unfortunately those were not enabled.. I'll build a new kernel with
>> CONFIG_DEBUGINFO enabled.
>>
>>     
>
> Actually CONFIG_DEBUG_INFO was enabled, if you meant that? 
>   

Yes, that's it.
> (gdb) x/i 0xc058cdcb
>   

Try "list *0xc058cdcb".

Thanks,
    J

Re: What is the current state of Dom0 kernel support? / crash

[Pasi Kärkkäinen]

On Wed, Jul 22, 2009 at 01:25:45PM -0700, Jeremy Fitzhardinge wrote:
> On 07/22/09 12:57, Pasi Kärkkäinen wrote:
> > On Wed, Jul 22, 2009 at 10:35:30PM +0300, Pasi Kärkkäinen wrote:
> >   
> >> On Wed, Jul 22, 2009 at 12:14:37PM -0700, Jeremy Fitzhardinge wrote:
> >>     
> >>> On 07/21/09 06:03, Pasi Kärkkäinen wrote:
> >>>       
> >>>> I just tried the latest 32b PAE rebase/master tree (2.6.31-rc3).
> >>>>
> >>>> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-10-rebase-master-with-highpte.txt
> >>>>
> >>>> Checking if this processor honours the WP bit even in supervisor mode...
> >>>> BUG: unable to handle kernel NULL pointer dereference at (null)
> >>>> IP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f
> >>>> *pdpt = 000000003d275001 
> >>>> Thread overran stack, or stack corrupted
> >>>> Oops: 0000 [#1] SMP 
> >>>> last sysfs file: 
> >>>> Modules linked in:
> >>>>
> >>>> Pid: 0, comm: swapper Not tainted (2.6.31-rc3 #20) P8SC8
> >>>> EIP: 0061:[<c058cdcb>] EFLAGS: 00010046 CPU: 0
> >>>> EIP is at xen_evtchn_do_upcall+0xcc/0x13f
> >>>> EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
> >>>> ESI: 00000000 EDI: c08ec558 EBP: c087eedc ESP: c087eea0
> >>>>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
> >>>> Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
> >>>> Stack:
> >>>>  00001a6e 00000220 00000200 00000000 00000000 00000000 e3201014 c08ec558
> >>>> <0> c087eee4 f5681000 e3201010 00000000 00000000 c09017f8 f54ff000 c087ef20
> >>>> <0> c0409927 00000000 c09017f8 f54ff000 c09017f8 f54ff000 c087ef20 c0843f70
> >>>> Call Trace:
> >>>>  [<c0409927>] ? xen_do_upcall+0x7/0xc
> >>>>  [<c0404581>] ? xen_pte_clear+0x9/0x12
> >>>>  [<c0427a94>] ? set_pte_vaddr+0xb4/0xc4
> >>>>  [<c0426c8c>] ? __native_set_fixmap+0x25/0x30
> >>>>  [<c040471a>] ? xen_set_fixmap+0xc7/0xcc
> >>>>  [<c0897d86>] ? mem_init+0x24a/0x298
> >>>>  [<c088367e>] ? start_kernel+0x14b/0x2cd
> >>>>  [<c088336f>] ? unknown_bootoption+0x0/0x18e
> >>>>  [<c0883082>] ? i386_start_kernel+0x71/0x79
> >>>>  [<c0886188>] ? xen_start_kernel+0x52a/0x533
> >>>> Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
> >>>> 8b 55 e4 e8 36 de e7 ff 8b 55 f0 8b 45 d0 03 
> >>>> 05 1c 0c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
> >>>> 18 
> >>>> EIP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
> >>>> CR2: 0000000000000000
> >>>> ---[ end trace 4eaa2a86a8e2da22 ]---
> >>>> Kernel panic - not syncing: Fatal exception in interrupt
> >>>>   
> >>>>         
> >>> Haven't seen that one before.  
> >>>
> >>>       
> >> Ok. I've seen many people report crashes during startup with rebase/master
> >> on 32b PAE. I assume they're seeing this same issue.
> >>
> >>     
> >>> The stack backtrace is a bit fuzzy; do you have CONFIG_FRAMEPOINTER enabled?  
> >>> And if you have CONFIG_DEBUGINFO enabled, you can map the eip c058cdcb 
> >>> to a specific source line (its not clear to me which pointer is NULL).
> >>>
> >>>       
> >> [root@dom0test linux-2.6-xen]# grep -i CONFIG_FRAMEPOINTER .config
> >> [root@dom0test linux-2.6-xen]# grep -i CONFIG_DEBUGINFO .config
> >> [root@dom0test linux-2.6-xen]# 
> >>
> >> Unfortunately those were not enabled.. I'll build a new kernel with
> >> CONFIG_DEBUGINFO enabled.
> >>
> >>     
> >
> > Actually CONFIG_DEBUG_INFO was enabled, if you meant that? 
> >   
> 
> Yes, that's it.
> > (gdb) x/i 0xc058cdcb
> >   
> 
> Try "list *0xc058cdcb".
> 

(gdb) list *0xc058cdcb
0xc058cdcb is in active_evtchns (drivers/xen/events.c:237).
232
233     static inline unsigned long active_evtchns(unsigned int cpu,
234                                                struct shared_info *sh,
235                                                unsigned int idx)
236     {
237             return (sh->evtchn_pending[idx] &
238                     cpu_evtchn_mask(cpu)[idx] &
239                     ~sh->evtchn_mask[idx]);
240     }
241
(gdb)


-- Pasi

Re: What is the current state of Dom0 kernel support?

[Anthony Wright]

Jeremy Fitzhardinge wrote:
> On 07/08/09 15:14, Pasi Kärkkäinen wrote:
>   
>> On Fri, Jun 26, 2009 at 11:29:30AM -0700, Jeremy Fitzhardinge wrote:
>>   
>>     
>>> On 06/26/09 11:21, Tim Post wrote:
>>>     
>>>       
>>>> Is it possible for you to set up a blog just for this? I think many
>>>> people are just going to pull your tree, it would be really, really nice
>>>> to have a feed to pull so we know when to pull and update .. especially
>>>> when the next merge window closes.
>>>>
>>>> As if you didn't have your hands full already :) Perhaps something on
>>>> xen.org just for kernel development?
>>>>
>>>> Sorry if something like this already exists.
>>>>   
>>>>       
>>>>         
>>> No, its a good idea.  I'll sort something out (and poke me if I don't
>>> appear to do anything in the next few days).
>>>
>>>     
>>>       
>> Btw what's the current tree people should be testing? xen-tip/next or
>> rebase/master? 
>>     
>
> rebase/master is what I'm currently working on.  It's work-in-progress,
> but it works for me at the moment.  I'd appreciate any test results you
> have.  (I don't yet have a fix in there for your PAE issue however.)
>
> I'm planning on renaming these branches to xen/... and proposing they
> become the basis for ongoing work
Jeremy, I'm desperately trying to move to a more up to date Dom0 kernel
as I'm finding it increasing difficult to find motherboards that work
with the drivers in 2.6.18 (Sometimes I can get away with patching the
kernel, but even this is a very poor solution because it means a
development & release cycle everytime somebody tries a new motherboard).

I've been following your attempts to mainline Dom0 support, and hoped
this would be the solution to my problems, but now that's been postponed
I'm trying to find an alternative solution.

I need a stable kernel as this is for production systems, and wondered
if you (or anybody else) could advise the best route to take.

I'm aware that you're continuing to work on the mainline patches and
wondered if you intend to stablise things in the next few months to
allow a formal replacement of the 2.6.18 Dom0 kernel. You seemed to be
suggesting this, but I wasn't quite sure. I also got the impression that
your patches don't have all the features of the 2.6.18 kernel, but again
I wasn't quite sure if this was the case, and if it was, whether it
would be a problem for most people.

I'm also a aware of a thread started by Kier a month or two ago about
replacing the 2.6.18 kernel with one of the rebased kernels, but wasn't
clear what conclusion was reached.

Instinctively I'd prefer to go with your patches, but failing that could
you/somebody recommend one of the rebased kernels.

thanks,
Anthony Wright.

Re: What is the current state of Dom0 kernel support?

[Pasi Kärkkäinen]

On Thu, Jul 23, 2009 at 09:22:57AM +0100, Anthony Wright wrote:
> Jeremy Fitzhardinge wrote:
> > On 07/08/09 15:14, Pasi Kärkkäinen wrote:
> >   
> >> On Fri, Jun 26, 2009 at 11:29:30AM -0700, Jeremy Fitzhardinge wrote:
> >>   
> >>     
> >>> On 06/26/09 11:21, Tim Post wrote:
> >>>     
> >>>       
> >>>> Is it possible for you to set up a blog just for this? I think many
> >>>> people are just going to pull your tree, it would be really, really nice
> >>>> to have a feed to pull so we know when to pull and update .. especially
> >>>> when the next merge window closes.
> >>>>
> >>>> As if you didn't have your hands full already :) Perhaps something on
> >>>> xen.org just for kernel development?
> >>>>
> >>>> Sorry if something like this already exists.
> >>>>   
> >>>>       
> >>>>         
> >>> No, its a good idea.  I'll sort something out (and poke me if I don't
> >>> appear to do anything in the next few days).
> >>>
> >>>     
> >>>       
> >> Btw what's the current tree people should be testing? xen-tip/next or
> >> rebase/master? 
> >>     
> >
> > rebase/master is what I'm currently working on.  It's work-in-progress,
> > but it works for me at the moment.  I'd appreciate any test results you
> > have.  (I don't yet have a fix in there for your PAE issue however.)
> >
> > I'm planning on renaming these branches to xen/... and proposing they
> > become the basis for ongoing work
> Jeremy, I'm desperately trying to move to a more up to date Dom0 kernel
> as I'm finding it increasing difficult to find motherboards that work
> with the drivers in 2.6.18 (Sometimes I can get away with patching the
> kernel, but even this is a very poor solution because it means a
> development & release cycle everytime somebody tries a new motherboard).
> 
> I've been following your attempts to mainline Dom0 support, and hoped
> this would be the solution to my problems, but now that's been postponed
> I'm trying to find an alternative solution.
> 
> I need a stable kernel as this is for production systems, and wondered
> if you (or anybody else) could advise the best route to take.
> 
> I'm aware that you're continuing to work on the mainline patches and
> wondered if you intend to stablise things in the next few months to
> allow a formal replacement of the 2.6.18 Dom0 kernel. You seemed to be
> suggesting this, but I wasn't quite sure. I also got the impression that
> your patches don't have all the features of the 2.6.18 kernel, but again
> I wasn't quite sure if this was the case, and if it was, whether it
> would be a problem for most people.
> 
> I'm also a aware of a thread started by Kier a month or two ago about
> replacing the 2.6.18 kernel with one of the rebased kernels, but wasn't
> clear what conclusion was reached.
> 
> Instinctively I'd prefer to go with your patches, but failing that could
> you/somebody recommend one of the rebased kernels.
> 

Please check this wiki page:
http://wiki.xensource.com/xenwiki/XenDom0Kernels

Hopefully it helps a bit.

pv_ops dom0 kernel was made the default in xen 3.5 development version
(xen-unstable). Xen 3.4.x still uses the old linux-2.6.18-xen tree as a
default.

I've been running pv_ops dom0 kernels for a while now, and the latest kernel
that worked for me (on 32bit PAE) was 2.6.30-rc6 from xen-tip/next tree.
although you need to disable CONFIG_HIGHPTE on 32bit, so make sure you have
CONFIG_HIGHPTE=n to work around a bug/race with xen_set_pte().

Current rebase/master doesn't boot on 32bit PAE, but it works on x86_64.

I bet and hope these issues will be sorted out shortly.. :)

-- Pasi

Re: What is the current state of Dom0 kernel support?

[Keir Fraser]

On 23/07/2009 09:22, "Anthony Wright" <anthony@overnetdata.com> wrote:
 
> I need a stable kernel as this is for production systems, and wondered
> if you (or anybody else) could advise the best route to take.

http://wiki.xensource.com/xenwiki/XenDom0Kernels

 -- Keir

Re: What is the current state of Dom0 kernel support?

[Simon Horman]

On Thu, Jul 23, 2009 at 09:37:44AM +0100, Keir Fraser wrote:
> On 23/07/2009 09:22, "Anthony Wright" <anthony@overnetdata.com> wrote:
>  
> > I need a stable kernel as this is for production systems, and wondered
> > if you (or anybody else) could advise the best route to take.
> 
> http://wiki.xensource.com/xenwiki/XenDom0Kernels

Is there a plan to release a dom0 kernel with 3.4.1 ?

Re: What is the current state of Dom0 kernel support? / crash

[Jeremy Fitzhardinge]

On 07/22/09 13:53, Pasi Kärkkäinen wrote:
> On Wed, Jul 22, 2009 at 01:25:45PM -0700, Jeremy Fitzhardinge wrote:
>   
>> On 07/22/09 12:57, Pasi Kärkkäinen wrote:
>>     
>>> On Wed, Jul 22, 2009 at 10:35:30PM +0300, Pasi Kärkkäinen wrote:
>>>   
>>>       
>>>> On Wed, Jul 22, 2009 at 12:14:37PM -0700, Jeremy Fitzhardinge wrote:
>>>>     
>>>>         
>>>>> On 07/21/09 06:03, Pasi Kärkkäinen wrote:
>>>>>       
>>>>>           
>>>>>> I just tried the latest 32b PAE rebase/master tree (2.6.31-rc3).
>>>>>>
>>>>>> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-10-rebase-master-with-highpte.txt
>>>>>>
>>>>>> Checking if this processor honours the WP bit even in supervisor mode...
>>>>>> BUG: unable to handle kernel NULL pointer dereference at (null)
>>>>>> IP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f
>>>>>> *pdpt = 000000003d275001 
>>>>>> Thread overran stack, or stack corrupted
>>>>>> Oops: 0000 [#1] SMP 
>>>>>> last sysfs file: 
>>>>>> Modules linked in:
>>>>>>
>>>>>> Pid: 0, comm: swapper Not tainted (2.6.31-rc3 #20) P8SC8
>>>>>> EIP: 0061:[<c058cdcb>] EFLAGS: 00010046 CPU: 0
>>>>>> EIP is at xen_evtchn_do_upcall+0xcc/0x13f
>>>>>> EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
>>>>>> ESI: 00000000 EDI: c08ec558 EBP: c087eedc ESP: c087eea0
>>>>>>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
>>>>>> Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
>>>>>> Stack:
>>>>>>  00001a6e 00000220 00000200 00000000 00000000 00000000 e3201014 c08ec558
>>>>>> <0> c087eee4 f5681000 e3201010 00000000 00000000 c09017f8 f54ff000 c087ef20
>>>>>> <0> c0409927 00000000 c09017f8 f54ff000 c09017f8 f54ff000 c087ef20 c0843f70
>>>>>> Call Trace:
>>>>>>  [<c0409927>] ? xen_do_upcall+0x7/0xc
>>>>>>  [<c0404581>] ? xen_pte_clear+0x9/0x12
>>>>>>  [<c0427a94>] ? set_pte_vaddr+0xb4/0xc4
>>>>>>  [<c0426c8c>] ? __native_set_fixmap+0x25/0x30
>>>>>>  [<c040471a>] ? xen_set_fixmap+0xc7/0xcc
>>>>>>  [<c0897d86>] ? mem_init+0x24a/0x298
>>>>>>  [<c088367e>] ? start_kernel+0x14b/0x2cd
>>>>>>  [<c088336f>] ? unknown_bootoption+0x0/0x18e
>>>>>>  [<c0883082>] ? i386_start_kernel+0x71/0x79
>>>>>>  [<c0886188>] ? xen_start_kernel+0x52a/0x533
>>>>>> Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
>>>>>> 8b 55 e4 e8 36 de e7 ff 8b 55 f0 8b 45 d0 03 
>>>>>> 05 1c 0c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
>>>>>> 18 
>>>>>> EIP: [<c058cdcb>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
>>>>>> CR2: 0000000000000000
>>>>>> ---[ end trace 4eaa2a86a8e2da22 ]---
>>>>>> Kernel panic - not syncing: Fatal exception in interrupt
>>>>>>   
>>>>>>         
>>>>>>             
>>>>> Haven't seen that one before.  
>>>>>
>>>>>       
>>>>>           
>>>> Ok. I've seen many people report crashes during startup with rebase/master
>>>> on 32b PAE. I assume they're seeing this same issue.
>>>>
>>>>     
>>>>         
>>>>> The stack backtrace is a bit fuzzy; do you have CONFIG_FRAMEPOINTER enabled?  
>>>>> And if you have CONFIG_DEBUGINFO enabled, you can map the eip c058cdcb 
>>>>> to a specific source line (its not clear to me which pointer is NULL).
>>>>>
>>>>>       
>>>>>           
>>>> [root@dom0test linux-2.6-xen]# grep -i CONFIG_FRAMEPOINTER .config
>>>> [root@dom0test linux-2.6-xen]# grep -i CONFIG_DEBUGINFO .config
>>>> [root@dom0test linux-2.6-xen]# 
>>>>
>>>> Unfortunately those were not enabled.. I'll build a new kernel with
>>>> CONFIG_DEBUGINFO enabled.
>>>>
>>>>     
>>>>         
>>> Actually CONFIG_DEBUG_INFO was enabled, if you meant that? 
>>>   
>>>       
>> Yes, that's it.
>>     
>>> (gdb) x/i 0xc058cdcb
>>>   
>>>       
>> Try "list *0xc058cdcb".
>>
>>     
>
> (gdb) list *0xc058cdcb
> 0xc058cdcb is in active_evtchns (drivers/xen/events.c:237).
> 232
> 233     static inline unsigned long active_evtchns(unsigned int cpu,
> 234                                                struct shared_info *sh,
> 235                                                unsigned int idx)
> 236     {
> 237             return (sh->evtchn_pending[idx] &
> 238                     cpu_evtchn_mask(cpu)[idx] &
> 239                     ~sh->evtchn_mask[idx]);
> 240     }
> 241
> (gdb)
>
>
> -- Pasi
>
>   

Does this help?

    J

Subject: [PATCH] xen: use proper percpu variable for cpu_evtchn_mask

cpu_evtchn_mask is a per-cpu mask of event channels, so it should
be implemented as a proper per_cpu variable.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index abad71b..4443b0f 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -93,14 +93,11 @@ static struct irq_info irq_info[NR_IRQS];
 static int evtchn_to_irq[NR_EVENT_CHANNELS] = {
 	[0 ... NR_EVENT_CHANNELS-1] = -1
 };
-struct cpu_evtchn_s {
-	unsigned long bits[NR_EVENT_CHANNELS/BITS_PER_LONG];
-};
-static struct cpu_evtchn_s *cpu_evtchn_mask_p;
-static inline unsigned long *cpu_evtchn_mask(int cpu)
-{
-	return cpu_evtchn_mask_p[cpu].bits;
-}
+
+#define NR_EVENT_CHANNEL_LONGS	(NR_EVENT_CHANNELS/BITS_PER_LONG)
+static DEFINE_PER_CPU(unsigned long,
+		      cpu_evtchn_mask[NR_EVENT_CHANNEL_LONGS]) =
+	{[0 ... NR_EVENT_CHANNEL_LONGS-1] = ~0};
 
 /* Xen will never allocate port zero for any purpose. */
 #define VALID_EVTCHN(chn)	((chn) != 0)
@@ -223,7 +220,7 @@ static inline unsigned long active_evtchns(unsigned int cpu,
 					   unsigned int idx)
 {
 	return (sh->evtchn_pending[idx] &
-		cpu_evtchn_mask(cpu)[idx] &
+		per_cpu(cpu_evtchn_mask, cpu)[idx] &
 		~sh->evtchn_mask[idx]);
 }
 
@@ -236,8 +233,8 @@ static void bind_evtchn_to_cpu(unsigned int chn, unsigned int cpu)
 	cpumask_copy(irq_to_desc(irq)->affinity, cpumask_of(cpu));
 #endif
 
-	__clear_bit(chn, cpu_evtchn_mask(cpu_from_irq(irq)));
-	__set_bit(chn, cpu_evtchn_mask(cpu));
+	__clear_bit(chn, per_cpu(cpu_evtchn_mask, cpu_from_irq(irq)));
+	__set_bit(chn, per_cpu(cpu_evtchn_mask, cpu));
 
 	irq_info[irq].cpu = cpu;
 }
@@ -253,8 +250,6 @@ static void init_evtchn_cpu_bindings(void)
 		cpumask_copy(desc->affinity, cpumask_of(0));
 	}
 #endif
-
-	memset(cpu_evtchn_mask(0), ~0, sizeof(cpu_evtchn_mask(0)));
 }
 
 static inline void clear_evtchn(int port)
@@ -928,10 +923,6 @@ void __init xen_init_IRQ(void)
 {
 	int i;
 
-	cpu_evtchn_mask_p = kcalloc(nr_cpu_ids, sizeof(struct cpu_evtchn_s),
-				    GFP_KERNEL);
-	BUG_ON(cpu_evtchn_mask_p == NULL);
-
 	init_evtchn_cpu_bindings();
 
 	/* No event channels are 'live' right now. */

Re: What is the current state of Dom0 kernel support? / crash

[Pasi Kärkkäinen]

On Wed, Jul 29, 2009 at 01:48:05PM -0700, Jeremy Fitzhardinge wrote:
> >>>>> The stack backtrace is a bit fuzzy; do you have CONFIG_FRAMEPOINTER enabled?  
> >>>>> And if you have CONFIG_DEBUGINFO enabled, you can map the eip c058cdcb 
> >>>>> to a specific source line (its not clear to me which pointer is NULL).
> >>>>>
> >>>>>       
> >>>>>           
> >>>> [root@dom0test linux-2.6-xen]# grep -i CONFIG_FRAMEPOINTER .config
> >>>> [root@dom0test linux-2.6-xen]# grep -i CONFIG_DEBUGINFO .config
> >>>> [root@dom0test linux-2.6-xen]# 
> >>>>
> >>>> Unfortunately those were not enabled.. I'll build a new kernel with
> >>>> CONFIG_DEBUGINFO enabled.
> >>>>
> >>>>     
> >>>>         
> >>> Actually CONFIG_DEBUG_INFO was enabled, if you meant that? 
> >>>   
> >>>       
> >> Yes, that's it.
> >>     
> >>> (gdb) x/i 0xc058cdcb
> >>>   
> >>>       
> >> Try "list *0xc058cdcb".
> >>
> >>     
> >
> > (gdb) list *0xc058cdcb
> > 0xc058cdcb is in active_evtchns (drivers/xen/events.c:237).
> > 232
> > 233     static inline unsigned long active_evtchns(unsigned int cpu,
> > 234                                                struct shared_info *sh,
> > 235                                                unsigned int idx)
> > 236     {
> > 237             return (sh->evtchn_pending[idx] &
> > 238                     cpu_evtchn_mask(cpu)[idx] &
> > 239                     ~sh->evtchn_mask[idx]);
> > 240     }
> > 241
> > (gdb)
> >
> >
> > -- Pasi
> >
> >   
> 
> Does this help?
> 

I'll try it later today. 

-- Pasi

>     J
> 
> Subject: [PATCH] xen: use proper percpu variable for cpu_evtchn_mask
> 
> cpu_evtchn_mask is a per-cpu mask of event channels, so it should
> be implemented as a proper per_cpu variable.
> 
> Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> 
> diff --git a/drivers/xen/events.c b/drivers/xen/events.c
> index abad71b..4443b0f 100644
> --- a/drivers/xen/events.c
> +++ b/drivers/xen/events.c
> @@ -93,14 +93,11 @@ static struct irq_info irq_info[NR_IRQS];
>  static int evtchn_to_irq[NR_EVENT_CHANNELS] = {
>  	[0 ... NR_EVENT_CHANNELS-1] = -1
>  };
> -struct cpu_evtchn_s {
> -	unsigned long bits[NR_EVENT_CHANNELS/BITS_PER_LONG];
> -};
> -static struct cpu_evtchn_s *cpu_evtchn_mask_p;
> -static inline unsigned long *cpu_evtchn_mask(int cpu)
> -{
> -	return cpu_evtchn_mask_p[cpu].bits;
> -}
> +
> +#define NR_EVENT_CHANNEL_LONGS	(NR_EVENT_CHANNELS/BITS_PER_LONG)
> +static DEFINE_PER_CPU(unsigned long,
> +		      cpu_evtchn_mask[NR_EVENT_CHANNEL_LONGS]) =
> +	{[0 ... NR_EVENT_CHANNEL_LONGS-1] = ~0};
>  
>  /* Xen will never allocate port zero for any purpose. */
>  #define VALID_EVTCHN(chn)	((chn) != 0)
> @@ -223,7 +220,7 @@ static inline unsigned long active_evtchns(unsigned int cpu,
>  					   unsigned int idx)
>  {
>  	return (sh->evtchn_pending[idx] &
> -		cpu_evtchn_mask(cpu)[idx] &
> +		per_cpu(cpu_evtchn_mask, cpu)[idx] &
>  		~sh->evtchn_mask[idx]);
>  }
>  
> @@ -236,8 +233,8 @@ static void bind_evtchn_to_cpu(unsigned int chn, unsigned int cpu)
>  	cpumask_copy(irq_to_desc(irq)->affinity, cpumask_of(cpu));
>  #endif
>  
> -	__clear_bit(chn, cpu_evtchn_mask(cpu_from_irq(irq)));
> -	__set_bit(chn, cpu_evtchn_mask(cpu));
> +	__clear_bit(chn, per_cpu(cpu_evtchn_mask, cpu_from_irq(irq)));
> +	__set_bit(chn, per_cpu(cpu_evtchn_mask, cpu));
>  
>  	irq_info[irq].cpu = cpu;
>  }
> @@ -253,8 +250,6 @@ static void init_evtchn_cpu_bindings(void)
>  		cpumask_copy(desc->affinity, cpumask_of(0));
>  	}
>  #endif
> -
> -	memset(cpu_evtchn_mask(0), ~0, sizeof(cpu_evtchn_mask(0)));
>  }
>  
>  static inline void clear_evtchn(int port)
> @@ -928,10 +923,6 @@ void __init xen_init_IRQ(void)
>  {
>  	int i;
>  
> -	cpu_evtchn_mask_p = kcalloc(nr_cpu_ids, sizeof(struct cpu_evtchn_s),
> -				    GFP_KERNEL);
> -	BUG_ON(cpu_evtchn_mask_p == NULL);
> -
>  	init_evtchn_cpu_bindings();
>  
>  	/* No event channels are 'live' right now. */
> 
> 

Re: What is the current state of Dom0 kernel support? / crash

[Pasi Kärkkäinen]

On Thu, Jul 30, 2009 at 11:48:00AM +0300, Pasi Kärkkäinen wrote:
> On Wed, Jul 29, 2009 at 01:48:05PM -0700, Jeremy Fitzhardinge wrote:
> > >>>>> The stack backtrace is a bit fuzzy; do you have CONFIG_FRAMEPOINTER enabled?  
> > >>>>> And if you have CONFIG_DEBUGINFO enabled, you can map the eip c058cdcb 
> > >>>>> to a specific source line (its not clear to me which pointer is NULL).
> > >>>>>
> > >>>>>       
> > >>>>>           
> > >>>> [root@dom0test linux-2.6-xen]# grep -i CONFIG_FRAMEPOINTER .config
> > >>>> [root@dom0test linux-2.6-xen]# grep -i CONFIG_DEBUGINFO .config
> > >>>> [root@dom0test linux-2.6-xen]# 
> > >>>>
> > >>>> Unfortunately those were not enabled.. I'll build a new kernel with
> > >>>> CONFIG_DEBUGINFO enabled.
> > >>>>
> > >>>>     
> > >>>>         
> > >>> Actually CONFIG_DEBUG_INFO was enabled, if you meant that? 
> > >>>   
> > >>>       
> > >> Yes, that's it.
> > >>     
> > >>> (gdb) x/i 0xc058cdcb
> > >>>   
> > >>>       
> > >> Try "list *0xc058cdcb".
> > >>
> > >>     
> > >
> > > (gdb) list *0xc058cdcb
> > > 0xc058cdcb is in active_evtchns (drivers/xen/events.c:237).
> > > 232
> > > 233     static inline unsigned long active_evtchns(unsigned int cpu,
> > > 234                                                struct shared_info *sh,
> > > 235                                                unsigned int idx)
> > > 236     {
> > > 237             return (sh->evtchn_pending[idx] &
> > > 238                     cpu_evtchn_mask(cpu)[idx] &
> > > 239                     ~sh->evtchn_mask[idx]);
> > > 240     }
> > > 241
> > > (gdb)
> > >
> > >
> > > -- Pasi
> > >
> > >   
> > 
> > Does this help?
> > 
> 
> I'll try it later today. 
> 

I did "git pull" to get the latest 2.6.31-rc4 tree, and the patch fails to
apply..

[root@dom0test linux-2.6-xen]# git pull

[root@dom0test linux-2.6-xen]# git reset --hard
HEAD is now at ef843c1 Merge commit 'v2.6.31-rc4' into rebase/master

[root@dom0test linux-2.6-xen]# patch -p1 < xen-pvops-dom0-use-proper-percpu-variable-for-cpu_evtchan_mask.patch
patching file drivers/xen/events.c
Hunk #1 FAILED at 93.
Hunk #2 succeeded at 232 (offset 12 lines).
Hunk #4 succeeded at 262 (offset 12 lines).
Hunk #5 FAILED at 935.
2 out of 5 hunks FAILED -- saving rejects to file drivers/xen/events.c.rej

Looking at the patch and comparing to my drivers/xen/events.c it seems not
the same version..

-- Pasi

Re: What is the current state of Dom0 kernel support? / crash

[Pasi Kärkkäinen]

On Thu, Jul 30, 2009 at 02:35:32PM +0300, Pasi Kärkkäinen wrote:
> > > >>>>         
> > > >>> Actually CONFIG_DEBUG_INFO was enabled, if you meant that? 
> > > >>>   
> > > >>>       
> > > >> Yes, that's it.
> > > >>     
> > > >>> (gdb) x/i 0xc058cdcb
> > > >>>   
> > > >>>       
> > > >> Try "list *0xc058cdcb".
> > > >>
> > > >>     
> > > >
> > > > (gdb) list *0xc058cdcb
> > > > 0xc058cdcb is in active_evtchns (drivers/xen/events.c:237).
> > > > 232
> > > > 233     static inline unsigned long active_evtchns(unsigned int cpu,
> > > > 234                                                struct shared_info *sh,
> > > > 235                                                unsigned int idx)
> > > > 236     {
> > > > 237             return (sh->evtchn_pending[idx] &
> > > > 238                     cpu_evtchn_mask(cpu)[idx] &
> > > > 239                     ~sh->evtchn_mask[idx]);
> > > > 240     }
> > > > 241
> > > > (gdb)
> > > >
> > > >
> > > > -- Pasi
> > > >
> > > >   
> > > 
> > > Does this help?
> > > 
> > 
> > I'll try it later today. 
> > 
> 
> I did "git pull" to get the latest 2.6.31-rc4 tree, and the patch fails to
> apply..
> 
> [root@dom0test linux-2.6-xen]# git pull
> 
> [root@dom0test linux-2.6-xen]# git reset --hard
> HEAD is now at ef843c1 Merge commit 'v2.6.31-rc4' into rebase/master
> 
> [root@dom0test linux-2.6-xen]# patch -p1 < xen-pvops-dom0-use-proper-percpu-variable-for-cpu_evtchan_mask.patch
> patching file drivers/xen/events.c
> Hunk #1 FAILED at 93.
> Hunk #2 succeeded at 232 (offset 12 lines).
> Hunk #4 succeeded at 262 (offset 12 lines).
> Hunk #5 FAILED at 935.
> 2 out of 5 hunks FAILED -- saving rejects to file drivers/xen/events.c.rej
> 
> Looking at the patch and comparing to my drivers/xen/events.c it seems not
> the same version..
> 

Did I mess up something here or was this patch against some other tree than
rebase/master? 

-- Pasi

Re: What is the current state of Dom0 kernel support? / crash

[Jeremy Fitzhardinge]

On 08/10/09 09:43, Pasi Kärkkäinen wrote:
> Did I mess up something here or was this patch against some other tree than
> rebase/master? 
>   

No, it needed another change in there.  rebase/master now has that fix
in it, but I'm seeing other interrupt-related strangeness which I still
don't understand.  I'd be interested in any reports you have about
current rebase/master.

    J

Re: What is the current state of Dom0 kernel support? / crash

[Pasi Kärkkäinen]

On Mon, Aug 10, 2009 at 09:52:35AM -0700, Jeremy Fitzhardinge wrote:
> On 08/10/09 09:43, Pasi Kärkkäinen wrote:
> > Did I mess up something here or was this patch against some other tree than
> > rebase/master? 
> >   
> 
> No, it needed another change in there.  rebase/master now has that fix
> in it, but I'm seeing other interrupt-related strangeness which I still
> don't understand.  I'd be interested in any reports you have about
> current rebase/master.
> 

I just updated rebase/master and rebuilt (now 2.6.31-rc5).

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-11-rebase-master-with-highpte.txt


Checking if this processor honours the WP bit even in supervisor mode...
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<c058c93f>] xen_evtchn_do_upcall+0xcc/0x13f
*pdpt = 000000003d276001 
Thread overran stack, or stack corrupted
Oops: 0000 [#1] SMP 
last sysfs file: 
Modules linked in:

Pid: 0, comm: swapper Not tainted (2.6.31-rc5 #21) P8SC8
EIP: 0061:[<c058c93f>] EFLAGS: 00010046 CPU: 0
EIP is at xen_evtchn_do_upcall+0xcc/0x13f
EAX: 00000000 EBX: ffffffff ECX: 00000000 EDX: 00000000
ESI: 00000000 EDI: c08ed558 EBP: c087eedc ESP: c087eea0
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
Process swapper (pid: 0, ti=c087e000 task=c083b1a0 task.ti=c087e000)
Stack:
 00001aa5 00000220 00000200 00000000 00000000 00000000 e3201014 c08ed558
<0> c087eee4 f5681000 e3201010 00000000 00000000 c09027f8 f54ff000 c087ef20
<0> c04099a7 00000000 c09027f8 f54ff000 c09027f8 f54ff000 c087ef20 c0843f70
Call Trace:
 [<c04099a7>] ? xen_do_upcall+0x7/0xc
 [<c0404591>] ? xen_pte_clear+0x9/0x12
 [<c0427bf8>] ? set_pte_vaddr+0xb4/0xc4
 [<c0426e80>] ? __native_set_fixmap+0x25/0x30
 [<c040472a>] ? xen_set_fixmap+0xc7/0xcc
 [<c0897f6c>] ? mem_init+0x24a/0x298
 [<c088367e>] ? start_kernel+0x14b/0x2cd
 [<c088336f>] ? unknown_bootoption+0x0/0x18e
 [<c0883082>] ? i386_start_kernel+0x71/0x79
 [<c088618d>] ? xen_start_kernel+0x52f/0x538
Code: d0 89 45 cc 89 55 c8 eb 16 0f bc c8 03 4d d4 8b 04 8a 83 f8 ff 74 f8
8b 55 e4 e8 42 e3 e7 ff 8b 55 f0 8b 45 d0 03 
05 1c 1c 97 c0 <8b> 0c 10 8b 55 e8 8b 45 cc 23 0c 82 8b 45 c8 8b 04 82 8b 15
18 
EIP: [<c058c93f>] xen_evtchn_do_upcall+0xcc/0x13f SS:ESP e021:c087eea0
CR2: 0000000000000000
---[ end trace 4eaa2a86a8e2da22 ]---
Kernel panic - not syncing: Fatal exception in interrupt



(gdb) list *0xc058c93f
0xc058c93f is in active_evtchns (drivers/xen/events.c:237).
232
233     static inline unsigned long active_evtchns(unsigned int cpu,
234                                                struct shared_info *sh,
235                                                unsigned int idx)
236     {
237             return (sh->evtchn_pending[idx] &
238                     cpu_evtchn_mask(cpu)[idx] &
239                     ~sh->evtchn_mask[idx]);
240     }
241
(gdb) 


-- Pasi

Re: What is the current state of Dom0 kernel support? / crash with 2.6.31-rc5 32bit PAE

[Pasi Kärkkäinen]

On Mon, Aug 10, 2009 at 10:27:50PM +0300, Pasi Kärkkäinen wrote:
> On Mon, Aug 10, 2009 at 09:52:35AM -0700, Jeremy Fitzhardinge wrote:
> > On 08/10/09 09:43, Pasi Kärkkäinen wrote:
> > > Did I mess up something here or was this patch against some other tree than
> > > rebase/master? 
> > >   
> > 
> > No, it needed another change in there.  rebase/master now has that fix
> > in it, but I'm seeing other interrupt-related strangeness which I still
> > don't understand.  I'd be interested in any reports you have about
> > current rebase/master.
> > 
> 
> I just updated rebase/master and rebuilt (now 2.6.31-rc5).
> 
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-11-rebase-master-with-highpte.txt
> 
> (gdb) list *0xc058c93f
> 0xc058c93f is in active_evtchns (drivers/xen/events.c:237).
> 232
> 233     static inline unsigned long active_evtchns(unsigned int cpu,
> 234                                                struct shared_info *sh,
> 235                                                unsigned int idx)
> 236     {
> 237             return (sh->evtchn_pending[idx] &
> 238                     cpu_evtchn_mask(cpu)[idx] &
> 239                     ~sh->evtchn_mask[idx]);
> 240     }
> 241
> (gdb) 
> 
> 

I just added some debug printk()'s to the very beginning of the function and found out this:

DEBUG active_evtchns: cpu: 0, sh: 4117237760, idx: 0
DEBUG sh->evtchn_pending: 4117239808, sh->evtchn_mask: 4117239936
DEBUG cpu_evtchn_mask(cpu): 0

so.. looks like the buggy piece of code (null pointer dereference) is: cpu_evtchn_mask(cpu)[idx]

-- Pasi

Re: What is the current state of Dom0 kernel support? / crash with 2.6.31-rc5 32bit PAE

[Jeremy Fitzhardinge]

On 08/11/09 12:58, Pasi Kärkkäinen wrote:
> On Mon, Aug 10, 2009 at 10:27:50PM +0300, Pasi Kärkkäinen wrote:
>   
>> On Mon, Aug 10, 2009 at 09:52:35AM -0700, Jeremy Fitzhardinge wrote:
>>     
>>> On 08/10/09 09:43, Pasi Kärkkäinen wrote:
>>>       
>>>> Did I mess up something here or was this patch against some other tree than
>>>> rebase/master? 
>>>>   
>>>>         
>>> No, it needed another change in there.  rebase/master now has that fix
>>> in it, but I'm seeing other interrupt-related strangeness which I still
>>> don't understand.  I'd be interested in any reports you have about
>>> current rebase/master.
>>>
>>>       
>> I just updated rebase/master and rebuilt (now 2.6.31-rc5).
>>
>> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-log-11-rebase-master-with-highpte.txt
>>
>> (gdb) list *0xc058c93f
>> 0xc058c93f is in active_evtchns (drivers/xen/events.c:237).
>> 232
>> 233     static inline unsigned long active_evtchns(unsigned int cpu,
>> 234                                                struct shared_info *sh,
>> 235                                                unsigned int idx)
>> 236     {
>> 237             return (sh->evtchn_pending[idx] &
>> 238                     cpu_evtchn_mask(cpu)[idx] &
>> 239                     ~sh->evtchn_mask[idx]);
>> 240     }
>> 241
>> (gdb) 
>>
>>
>>     
>
> I just added some debug printk()'s to the very beginning of the function and found out this:
>
> DEBUG active_evtchns: cpu: 0, sh: 4117237760, idx: 0
> DEBUG sh->evtchn_pending: 4117239808, sh->evtchn_mask: 4117239936
> DEBUG cpu_evtchn_mask(cpu): 0
>
> so.. looks like the buggy piece of code (null pointer dereference) is: cpu_evtchn_mask(cpu)[idx]
>   

Yes; something is trying to use that stuff before the array is
allocated.  I'm trying to work out a fix now, and I wonder if its
related to some (more subtle) interrupt-related problems I'm seeing.

    J

Re: What is the current state of Dom0 kernel support? / crash with 2.6.31-rc5 32bit PAE

[Jeremy Fitzhardinge]

On 08/11/09 12:58, Pasi Kärkkäinen wrote:
> I just added some debug printk()'s to the very beginning of the function and found out this:
>
> DEBUG active_evtchns: cpu: 0, sh: 4117237760, idx: 0
> DEBUG sh->evtchn_pending: 4117239808, sh->evtchn_mask: 4117239936
> DEBUG cpu_evtchn_mask(cpu): 0
>
> so.. looks like the buggy piece of code (null pointer dereference) is: cpu_evtchn_mask(cpu)[idx]
>
>   

I pushed something that looks like a fix for this.  Please tell me how
it goes.

    J

Re: What is the current state of Dom0 kernel support? / crash with 2.6.31-rc5 32bit PAE

[Pasi Kärkkäinen]

On Thu, Aug 13, 2009 at 01:17:40PM -0700, Jeremy Fitzhardinge wrote:
> On 08/11/09 12:58, Pasi Kärkkäinen wrote:
> > I just added some debug printk()'s to the very beginning of the function and found out this:
> >
> > DEBUG active_evtchns: cpu: 0, sh: 4117237760, idx: 0
> > DEBUG sh->evtchn_pending: 4117239808, sh->evtchn_mask: 4117239936
> > DEBUG cpu_evtchn_mask(cpu): 0
> >
> > so.. looks like the buggy piece of code (null pointer dereference) is: cpu_evtchn_mask(cpu)[idx]
> >
> >   
> 
> I pushed something that looks like a fix for this.  Please tell me how
> it goes.
> 

Ok. I can only test next tuesday. I'll let you know then.

-- Pasi

Re: What is the current state of Dom0 kernel support? / 2.6.31-rc5 32bit PAE works now!

[Pasi Kärkkäinen]

On Fri, Aug 14, 2009 at 05:15:38PM +0300, Pasi Kärkkäinen wrote:
> On Thu, Aug 13, 2009 at 01:17:40PM -0700, Jeremy Fitzhardinge wrote:
> > On 08/11/09 12:58, Pasi Kärkkäinen wrote:
> > > I just added some debug printk()'s to the very beginning of the function and found out this:
> > >
> > > DEBUG active_evtchns: cpu: 0, sh: 4117237760, idx: 0
> > > DEBUG sh->evtchn_pending: 4117239808, sh->evtchn_mask: 4117239936
> > > DEBUG cpu_evtchn_mask(cpu): 0
> > >
> > > so.. looks like the buggy piece of code (null pointer dereference) is: cpu_evtchn_mask(cpu)[idx]
> > >
> > >   
> > 
> > I pushed something that looks like a fix for this.  Please tell me how
> > it goes.
> > 
> 

.. and now it boots just fine, no crashes anymore!

Thanks a lot!

-- Pasi

Re: What is the current state of Dom0 kernel support? / 2.6.31-rc5 32bit PAE works now!

[Pasi Kärkkäinen]

On Mon, Aug 17, 2009 at 06:04:55PM +0300, Pasi Kärkkäinen wrote:
> On Fri, Aug 14, 2009 at 05:15:38PM +0300, Pasi Kärkkäinen wrote:
> > On Thu, Aug 13, 2009 at 01:17:40PM -0700, Jeremy Fitzhardinge wrote:
> > > On 08/11/09 12:58, Pasi Kärkkäinen wrote:
> > > > I just added some debug printk()'s to the very beginning of the function and found out this:
> > > >
> > > > DEBUG active_evtchns: cpu: 0, sh: 4117237760, idx: 0
> > > > DEBUG sh->evtchn_pending: 4117239808, sh->evtchn_mask: 4117239936
> > > > DEBUG cpu_evtchn_mask(cpu): 0
> > > >
> > > > so.. looks like the buggy piece of code (null pointer dereference) is: cpu_evtchn_mask(cpu)[idx]
> > > >
> > > >   
> > > 
> > > I pushed something that looks like a fix for this.  Please tell me how
> > > it goes.
> > > 
> > 
> 
> .. and now it boots just fine, no crashes anymore!
> 
> Thanks a lot!
> 

Just for the record I'm still using CONFIG_HIGHPTE=n since I believe that is
still broken on 32bit.

-- Pasi

Re: What is the current state of Dom0 kernel support? / 2.6.31-rc5 32bit PAE works now!

[Jeremy Fitzhardinge]

On 08/17/09 08:26, Pasi Kärkkäinen wrote:
>> .. and now it boots just fine, no crashes anymore!
>>
>> Thanks a lot!
>>     

Ah, good.

> Just for the record I'm still using CONFIG_HIGHPTE=n since I believe that is
> still broken on 32bit.
>   

Yes, it is.  HIGHPTE isn't recommended in general because it imposes a
fairly high performance hit (even without Xen, but worse with).  And,
worse, at the moment I don't have any plausible fix for the bug.  Its
really a left-over from the days of 32-bit machines with much more than
4G of memory; these days you'd use 64-bit.

    J