Announcing xen/master: pvops git trees rearranged

Announcing xen/master: pvops git trees rearranged

[Jeremy Fitzhardinge]

Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
was a fairly nasty bug which ended up corrupting dom0 memory when doing
IO on behalf of domains, but that is finally fixed.

In honour of this, I've renamed the rebase/* branches to xen/* (moving
the old remaining xen/* branches to xen-old/*); xen/master is now the
preferred branch for fetching current dom0 work.

The kernel tree is fairly featureful:

    * basic dom0 support
    * blkback
    * netback
    * ACPI power management
    * S3 suspend/resume (at least for some people)
    * microcode update
    * MSI support

In other words, it has as much as it ever has.

There are a few notable missing features:

    * blktap2
    * netchannel2
    * pci front/back
    * upstream Linux support
    * your pet feature


Full coordinates are:

    git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git xen/master

See http://wiki.xensource.com/xenwiki/XenParavirtOps for general
directions on configuration, compilation and use.

This is definitely a work-in-progress kernel.  I'd appreciate all bug
*and* success reports so I can get some idea of how many people are
using this thing, and how often there are problems.  Patches gratefully
accepted.

Thanks,
    J

Re: Announcing xen/master: pvops git trees rearranged

[Marc - A. Dahlhaus]

Hello Jeremy,

Jeremy Fitzhardinge schrieb:
> Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
> was a fairly nasty bug which ended up corrupting dom0 memory when doing
> IO on behalf of domains, but that is finally fixed.
> 
> In honour of this, I've renamed the rebase/* branches to xen/* (moving
> the old remaining xen/* branches to xen-old/*); xen/master is now the
> preferred branch for fetching current dom0 work.
> 
> The kernel tree is fairly featureful:
> 
>     * basic dom0 support
>     * blkback
>     * netback
>     * ACPI power management
>     * S3 suspend/resume (at least for some people)
>     * microcode update
>     * MSI support
> 
> In other words, it has as much as it ever has.

waited for MSI, will give it a try :o)

> There are a few notable missing features:
> 
>     * blktap2
>     * netchannel2
>     * pci front/back
>     * upstream Linux support
>     * your pet feature
> 
> 
> Full coordinates are:
> 
>     git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git xen/master

Is HEAD already pointing to xen/master or do i have to change branches?

> See http://wiki.xensource.com/xenwiki/XenParavirtOps for general
> directions on configuration, compilation and use.

I think it would be a good idea to add some backtrace and debug related 
options to the CONFIG_ options listed on the wiki page so that one gets 
all the details needed from the testers when some bug shows up (CCd 
Pasi). The NETXEN_NIC option looks unrelated to xen...

> This is definitely a work-in-progress kernel.  I'd appreciate all bug
> *and* success reports so I can get some idea of how many people are
> using this thing, and how often there are problems.  Patches gratefully
> accepted.

Will report something back soon,


Marc

Re: Announcing xen/master: pvops git trees rearranged

[Marc - A. Dahlhaus]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

Marc - A. Dahlhaus schrieb:
> Hello Jeremy,
> 
> Jeremy Fitzhardinge schrieb:
>> Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
>> was a fairly nasty bug which ended up corrupting dom0 memory when doing
>> IO on behalf of domains, but that is finally fixed.
>>
>> In honour of this, I've renamed the rebase/* branches to xen/* (moving
>> the old remaining xen/* branches to xen-old/*); xen/master is now the
>> preferred branch for fetching current dom0 work.
>>
--8<--
>> See http://wiki.xensource.com/xenwiki/XenParavirtOps for general
>> directions on configuration, compilation and use.
> 
> I think it would be a good idea to add some backtrace and debug related 
> options to the CONFIG_ options listed on the wiki page so that one gets 
> all the details needed from the testers when some bug shows up (CCd 
> Pasi). The NETXEN_NIC option looks unrelated to xen...

Somehow my mail gateway got problems with umlauts and rewrote 
mailadress, so this is another try to CC pasi...

>> This is definitely a work-in-progress kernel.  I'd appreciate all bug
>> *and* success reports so I can get some idea of how many people are
>> using this thing, and how often there are problems.  Patches gratefully
>> accepted.
> 
> Will report something back soon,

Had no luck. It crashed during irq initialization. Somehow the serial 
logs aren't fully transmitted. All i could catch is attached. Also is 
attached the .config used to build the dom0 kernel. Will give xen 3.4.1 
and unstable another try tomorrow.

Marc

[-- Attachment #2: xen.log.gz --]
[-- Type: application/x-gzip, Size: 7968 bytes --]

[-- Attachment #3: config-2.6.31-xen.gz --]
[-- Type: application/x-gzip, Size: 16838 bytes --]

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Sun, Sep 20, 2009 at 09:29:46PM +0200, Marc - A. Dahlhaus wrote:
> Marc - A. Dahlhaus schrieb:
> >Hello Jeremy,
> >
> >Jeremy Fitzhardinge schrieb:
> >>Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
> >>was a fairly nasty bug which ended up corrupting dom0 memory when doing
> >>IO on behalf of domains, but that is finally fixed.
> >>
> >>In honour of this, I've renamed the rebase/* branches to xen/* (moving
> >>the old remaining xen/* branches to xen-old/*); xen/master is now the
> >>preferred branch for fetching current dom0 work.
> >>
> --8<--
> >>See http://wiki.xensource.com/xenwiki/XenParavirtOps for general
> >>directions on configuration, compilation and use.
> >
> >I think it would be a good idea to add some backtrace and debug related 
> >options to the CONFIG_ options listed on the wiki page so that one gets 
> >all the details needed from the testers when some bug shows up (CCd 
> >Pasi). The NETXEN_NIC option looks unrelated to xen...
> 
> Somehow my mail gateway got problems with umlauts and rewrote 
> mailadress, so this is another try to CC pasi...
>

Heh. It's kinda funny to have all these charset/umlaut problems still in 2009 :)

> >>This is definitely a work-in-progress kernel.  I'd appreciate all bug
> >>*and* success reports so I can get some idea of how many people are
> >>using this thing, and how often there are problems.  Patches gratefully
> >>accepted.
> >
> >Will report something back soon,
> 
> Had no luck. It crashed during irq initialization. Somehow the serial 
> logs aren't fully transmitted. All i could catch is attached. Also is 
> attached the .config used to build the dom0 kernel. Will give xen 3.4.1 
> and unstable another try tomorrow.
> 

Please paste your grub.conf. I'm going to try xen/master tree today..

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Marc - A. Dahlhaus [ Administration | Westermann GmbH ]]

Am Montag, den 21.09.2009, 09:22 +0300 schrieb Pasi Kärkkäinen:
> On Sun, Sep 20, 2009 at 09:29:46PM +0200, Marc - A. Dahlhaus wrote:
> > Marc - A. Dahlhaus schrieb:
> > >Hello Jeremy,
> > >
> > >Jeremy Fitzhardinge schrieb:
> > >>Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
> > >>was a fairly nasty bug which ended up corrupting dom0 memory when doing
> > >>IO on behalf of domains, but that is finally fixed.
> > >>
> > >>In honour of this, I've renamed the rebase/* branches to xen/* (moving
> > >>the old remaining xen/* branches to xen-old/*); xen/master is now the
> > >>preferred branch for fetching current dom0 work.
> > >>
> > --8<--
> > >>See http://wiki.xensource.com/xenwiki/XenParavirtOps for general
> > >>directions on configuration, compilation and use.
> > >
> > >I think it would be a good idea to add some backtrace and debug related 
> > >options to the CONFIG_ options listed on the wiki page so that one gets 
> > >all the details needed from the testers when some bug shows up (CCd 
> > >Pasi). The NETXEN_NIC option looks unrelated to xen...
> > 
> > Somehow my mail gateway got problems with umlauts and rewrote 
> > mailadress, so this is another try to CC pasi...
> >
> 
> Heh. It's kinda funny to have all these charset/umlaut problems still in 2009 :)

Stange problem as we don't had such problems with other mail addresses
that used umlauts in headers, we're working on reproducing it. Looks
like a Perl-Module related problem in the spam filter as far as we got
until now...

> > >>This is definitely a work-in-progress kernel.  I'd appreciate all bug
> > >>*and* success reports so I can get some idea of how many people are
> > >>using this thing, and how often there are problems.  Patches gratefully
> > >>accepted.
> > >
> > >Will report something back soon,
> > 
> > Had no luck. It crashed during irq initialization. Somehow the serial 
> > logs aren't fully transmitted. All i could catch is attached. Also is 
> > attached the .config used to build the dom0 kernel. Will give xen 3.4.1 
> > and unstable another try tomorrow.
> > 
> 
> Please paste your grub.conf. I'm going to try xen/master tree today..
> 
> -- Pasi

title XEN pv_ops dom0
root (hd0,1)
kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT earlyprintk=xen 4
module /boot/initramfs-2.6.31-xen

With this one i get no output from the kernel itself at all, i think i
have some things build as modules in my .config that needs to be buildin
for a working vga console (might be that i need to change the
earlyprintk param to vga) but didn't had the time to hunt the problem
down.


title XEN pv_ops dom0 with serial console
root (hd0,1)
kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
com1=9600,8n1 console=com1
module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT console=ttyS0,9600n8
earlyprintk=xen 4
module /boot/initramfs-2.6.31-xen

That's the one i used to grab the log.

This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram installed
but only 2912584k usable because of an PCI-Card that supports only 32bit
addressing (appropriated-ram-crap). Could this be a problem? I try to
test again with 2GB Ram installed this evening and report if it gets any
further...


Marc

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Mon, Sep 21, 2009 at 10:49:03AM +0200, Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:
> > > >>This is definitely a work-in-progress kernel.  I'd appreciate all bug
> > > >>*and* success reports so I can get some idea of how many people are
> > > >>using this thing, and how often there are problems.  Patches gratefully
> > > >>accepted.
> > > >
> > > >Will report something back soon,
> > > 
> > > Had no luck. It crashed during irq initialization. Somehow the serial 
> > > logs aren't fully transmitted. All i could catch is attached. Also is 
> > > attached the .config used to build the dom0 kernel. Will give xen 3.4.1 
> > > and unstable another try tomorrow.
> > > 
> > 
> > Please paste your grub.conf. I'm going to try xen/master tree today..
> > 
> > -- Pasi
> 
> title XEN pv_ops dom0
> root (hd0,1)
> kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT earlyprintk=xen 4
> module /boot/initramfs-2.6.31-xen
> 
> With this one i get no output from the kernel itself at all, i think i
> have some things build as modules in my .config that needs to be buildin
> for a working vga console (might be that i need to change the
> earlyprintk param to vga) but didn't had the time to hunt the problem
> down.
> 
> 
> title XEN pv_ops dom0 with serial console
> root (hd0,1)
> kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> com1=9600,8n1 console=com1
> module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT console=ttyS0,9600n8
> earlyprintk=xen 4
> module /boot/initramfs-2.6.31-xen
> 
> That's the one i used to grab the log.
> 
> This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram installed
> but only 2912584k usable because of an PCI-Card that supports only 32bit
> addressing (appropriated-ram-crap). Could this be a problem? I try to
> test again with 2GB Ram installed this evening and report if it gets any
> further...
> 

Here are my working grub.conf entries for pv_ops dom0:

VGA text console:

title Fedora Xen pv_ops dom0-test (2.6.31-rc5)
        root (hd0,0)
        kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
        module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01
        module /initrd-2.6.31-rc5.img


Serial console:

title Fedora Xen pv_ops dom0-test (2.6.31-rc5) / serial console
        root (hd0,0)
        kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all com1=19200,8n1 console=com1
        module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01 console=hvc0 earlyprintk=xen
        module /initrd-2.6.31-rc5.img

When using serial console with Xen you don't need to specify serial console stuff for the dom0 kernel..
Also note the 'console=hvc0' for dom0 kernel..

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Marc - A. Dahlhaus [ Administration | Westermann GmbH ]]

Am Montag, den 21.09.2009, 12:03 +0300 schrieb Pasi Kärkkäinen:
> On Mon, Sep 21, 2009 at 10:49:03AM +0200, Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:
--8<--
> > title XEN pv_ops dom0
> > root (hd0,1)
> > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT earlyprintk=xen 4
> > module /boot/initramfs-2.6.31-xen
> > 
> > With this one i get no output from the kernel itself at all, i think i
> > have some things build as modules in my .config that needs to be buildin
> > for a working vga console (might be that i need to change the
> > earlyprintk param to vga) but didn't had the time to hunt the problem
> > down.
> > 
> > 
> > title XEN pv_ops dom0 with serial console
> > root (hd0,1)
> > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > com1=9600,8n1 console=com1
> > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT console=ttyS0,9600n8
> > earlyprintk=xen 4
> > module /boot/initramfs-2.6.31-xen
> > 
> > That's the one i used to grab the log.
> > 
> > This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram installed
> > but only 2912584k usable because of an PCI-Card that supports only 32bit
> > addressing (appropriated-ram-crap). Could this be a problem? I try to
> > test again with 2GB Ram installed this evening and report if it gets any
> > further...
> > 
> 
> Here are my working grub.conf entries for pv_ops dom0:
> 
> VGA text console:
> 
> title Fedora Xen pv_ops dom0-test (2.6.31-rc5)
>         root (hd0,0)
>         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
>         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01
>         module /initrd-2.6.31-rc5.img
> 
> 
> Serial console:
> 
> title Fedora Xen pv_ops dom0-test (2.6.31-rc5) / serial console
>         root (hd0,0)
>         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all com1=19200,8n1 console=com1
>         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01 console=hvc0 earlyprintk=xen
>         module /initrd-2.6.31-rc5.img
> 
> When using serial console with Xen you don't need to specify serial console stuff for the dom0 kernel..
> Also note the 'console=hvc0' for dom0 kernel..

Isn't the selection of hvc0 as main console supposed to work out
automatically? Looks like a bug if it isn't. Anyway thanks for the hint.
Will give it a try with a grub config adapted to yours later today.

Are there still problems related to specific CONFIG_ options which i
should deactivate in my build?

The xen related options i used (only frontends build as modules):

[mad@darkstar ~]$ grep XEN /boot/config-2.6.31-xen
CONFIG_XEN=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=8
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_DOM0_PCI=y
CONFIG_MICROCODE_XEN=y
CONFIG_PCI_XEN=y
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_KBDDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=m
CONFIG_XEN_S3=y
CONFIG_ACPI_PROCESSOR_XEN=y


Marc

Re: Announcing xen/master: pvops git trees rearranged

[Konrad Rzeszutek Wilk]

> > > title XEN pv_ops dom0 with serial console
> > > root (hd0,1)
> > > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > > com1=9600,8n1 console=com1
> > > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT console=ttyS0,9600n8
> > > earlyprintk=xen 4

Do also add 'debug' in the line. That will give some more verbose output.

> > > module /boot/initramfs-2.6.31-xen
> > > 
> > > That's the one i used to grab the log.
> > > 
> > > This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram installed
> > > but only 2912584k usable because of an PCI-Card that supports only 32bit
> > > addressing (appropriated-ram-crap). Could this be a problem? I try to
> > > test again with 2GB Ram installed this evening and report if it gets any
> > > further...

If you specify 'mem=2GB' it will limit how much memory Xen and Linux see.
But you do use 'dom0_mem' which limits the amount of memory that
Dom0 would use for PCI drivers. So the 32-bit support shouldn't be an issue.

Looking at the serial line it also looks that your Linux is 32-bit? Did you
try the 64-bit?

> > > 
> > 
> > Here are my working grub.conf entries for pv_ops dom0:
> > 
> > VGA text console:
> > 
> > title Fedora Xen pv_ops dom0-test (2.6.31-rc5)
> >         root (hd0,0)
> >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
> >         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01
> >         module /initrd-2.6.31-rc5.img
> > 
> > 
> > Serial console:
> > 
> > title Fedora Xen pv_ops dom0-test (2.6.31-rc5) / serial console
> >         root (hd0,0)
> >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all com1=19200,8n1 console=com1
> >         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01 console=hvc0 earlyprintk=xen
> >         module /initrd-2.6.31-rc5.img
> > 
> > When using serial console with Xen you don't need to specify serial console stuff for the dom0 kernel..
> > Also note the 'console=hvc0' for dom0 kernel..
> 
> Isn't the selection of hvc0 as main console supposed to work out
> automatically? Looks like a bug if it isn't. Anyway thanks for the hint.

Yes. As long as CONFIG_HVC_XEN is compiled in (=y).

> Will give it a try with a grub config adapted to yours later today.
> 
> Are there still problems related to specific CONFIG_ options which i
> should deactivate in my build?

It should be all clear. But we haven't tested all of them, so you might be hitting
one that hasn't been tested yet.

[Cluster-devel] Re: [Xen-devel] Announcing xen/master: pvops git trees rearranged

[Marc]

Am Montag, den 21.09.2009, 10:39 -0400 schrieb Konrad Rzeszutek Wilk:
> > > > title XEN pv_ops dom0 with serial console
> > > > root (hd0,1)
> > > > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > > > com1=9600,8n1 console=com1
> > > > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT console=ttyS0,9600n8
> > > > earlyprintk=xen 4
> 
> Do also add 'debug' in the line. That will give some more verbose output.

Will give it a try.

> > > > module /boot/initramfs-2.6.31-xen
> > > > 
> > > > That's the one i used to grab the log.
> > > > 
> > > > This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram installed
> > > > but only 2912584k usable because of an PCI-Card that supports only 32bit
> > > > addressing (appropriated-ram-crap). Could this be a problem? I try to
> > > > test again with 2GB Ram installed this evening and report if it gets any
> > > > further...
> 
> If you specify 'mem=2GB' it will limit how much memory Xen and Linux see.
> But you do use 'dom0_mem' which limits the amount of memory that
> Dom0 would use for PCI drivers. So the 32-bit support shouldn't be an issue.
> 
> Looking at the serial line it also looks that your Linux is 32-bit?

Thats right (xen and dom0 are 32bit on that box so far).

> Did you try the 64-bit?

No. As i use GFS (cluster filesystem) and some hosts are 32bit only in
my little home cluster it didn't make sense so far to switch to 64bit.

Would make sense if dom0 would boot on this hosts as i could switch the
cluster-node on this box into a 32bit domU...

> > > Here are my working grub.conf entries for pv_ops dom0:
> > > 
> > > VGA text console:
> > > 
> > > title Fedora Xen pv_ops dom0-test (2.6.31-rc5)
> > >         root (hd0,0)
> > >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
> > >         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01
> > >         module /initrd-2.6.31-rc5.img
> > > 
> > > 
> > > Serial console:
> > > 
> > > title Fedora Xen pv_ops dom0-test (2.6.31-rc5) / serial console
> > >         root (hd0,0)
> > >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all com1=19200,8n1 console=com1
> > >         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01 console=hvc0 earlyprintk=xen
> > >         module /initrd-2.6.31-rc5.img
> > > 
> > > When using serial console with Xen you don't need to specify serial console stuff for the dom0 kernel..
> > > Also note the 'console=hvc0' for dom0 kernel..
> > 
> > Isn't the selection of hvc0 as main console supposed to work out
> > automatically? Looks like a bug if it isn't. Anyway thanks for the hint.
> 
> Yes. As long as CONFIG_HVC_XEN is compiled in (=y).

Thanks for confirmation.

> > Will give it a try with a grub config adapted to yours later today.
> > 
> > Are there still problems related to specific CONFIG_ options which i
> > should deactivate in my build?
> 
> It should be all clear. But we haven't tested all of them, so you might be hitting
> one that hasn't been tested yet.

Thanks,


Marc

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Mon, Sep 21, 2009 at 11:18:27AM +0200, Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:
> Am Montag, den 21.09.2009, 12:03 +0300 schrieb Pasi Kärkkäinen:
> > On Mon, Sep 21, 2009 at 10:49:03AM +0200, Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:
> --8<--
> > > title XEN pv_ops dom0
> > > root (hd0,1)
> > > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT earlyprintk=xen 4
> > > module /boot/initramfs-2.6.31-xen
> > > 
> > > With this one i get no output from the kernel itself at all, i think i
> > > have some things build as modules in my .config that needs to be buildin
> > > for a working vga console (might be that i need to change the
> > > earlyprintk param to vga) but didn't had the time to hunt the problem
> > > down.
> > > 
> > > 
> > > title XEN pv_ops dom0 with serial console
> > > root (hd0,1)
> > > kernel /boot/xen.gz dom0_mem=512M loglvl=all guest_loglvl=all
> > > com1=9600,8n1 console=com1
> > > module /boot/bzImage-2.6.31-xen ro root=LABEL=ROOT console=ttyS0,9600n8
> > > earlyprintk=xen 4
> > > module /boot/initramfs-2.6.31-xen
> > > 
> > > That's the one i used to grab the log.
> > > 
> > > This is a i945 based box with an Core 2 Duo E8400 and 4GB Ram installed
> > > but only 2912584k usable because of an PCI-Card that supports only 32bit
> > > addressing (appropriated-ram-crap). Could this be a problem? I try to
> > > test again with 2GB Ram installed this evening and report if it gets any
> > > further...
> > > 
> > 
> > Here are my working grub.conf entries for pv_ops dom0:
> > 
> > VGA text console:
> > 
> > title Fedora Xen pv_ops dom0-test (2.6.31-rc5)
> >         root (hd0,0)
> >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all
> >         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01
> >         module /initrd-2.6.31-rc5.img
> > 
> > 
> > Serial console:
> > 
> > title Fedora Xen pv_ops dom0-test (2.6.31-rc5) / serial console
> >         root (hd0,0)
> >         kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all com1=19200,8n1 console=com1
> >         module /vmlinuz-2.6.31-rc5 ro root=/dev/mapper/vg_dom0test-lv01 console=hvc0 earlyprintk=xen
> >         module /initrd-2.6.31-rc5.img
> > 
> > When using serial console with Xen you don't need to specify serial console stuff for the dom0 kernel..
> > Also note the 'console=hvc0' for dom0 kernel..
> 
> Isn't the selection of hvc0 as main console supposed to work out
> automatically? Looks like a bug if it isn't. Anyway thanks for the hint.
> Will give it a try with a grub config adapted to yours later today.
> 
> Are there still problems related to specific CONFIG_ options which i
> should deactivate in my build?
> 

CONFIG_HIGHPTE=y makes dom0 kernel crash for me.. there's a known race
and a hacky patch available, but no clean solution..

CONFIG_HIGHPTE=n is stable for me.

This is on 32bit PAE. 

-- Pasi

pvops: AHCI problems with SB600

[Patrick Scharrenberg]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

Hi,

I just tested this branch and found that the system gets stuck on
initializing the AHCI controller (AMD SB600).
Please find the error messages attached as png (unfortunately netconsole
doesn't work either)

When booting the same kernel natively without xen, all works fine
(including netconsole).


Patrick

[-- Attachment #2: sb600.png --]
[-- Type: image/png, Size: 24486 bytes --]

[-- Attachment #3: lspci-nvnv --]
[-- Type: text/plain, Size: 14520 bytes --]

00:00.0 Host bridge [0600]: ATI Technologies Inc RS690 Host Bridge [1002:7910]
	Subsystem: ASUSTeK Computer Inc. Device [1043:826d]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
	Latency: 64

00:01.0 PCI bridge [0604]: ATI Technologies Inc RS690 PCI to PCI Bridge (Internal gfx) [1002:7912] (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 99
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=68
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: fdb00000-fdcfffff
	Prefetchable memory behind bridge: 00000000fa000000-00000000fbffffff
	Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [44] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [b0] Subsystem: ATI Technologies Inc RS690 PCI to PCI Bridge (Internal gfx) [1002:7912]

00:07.0 PCI bridge [0604]: ATI Technologies Inc RS690 PCI to PCI Bridge (PCI Express Port 3) [1002:7917] (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: fda00000-fdafffff
	Prefetchable memory behind bridge: 00000000fdf00000-00000000fdffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v1) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Suprise- LLActRep+ BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
	Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable-
		Address: 00000000  Data: 0000
	Capabilities: [b0] Subsystem: ASUSTeK Computer Inc. Device [1043:826d]
	Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+

00:12.0 SATA controller [0106]: ATI Technologies Inc SB600 Non-Raid-5 SATA [1002:4380] (prog-if 01 [AHCI 1.0])
	Subsystem: ASUSTeK Computer Inc. Device [1043:8231]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64
	Interrupt: pin A routed to IRQ 16
	Region 0: I/O ports at ff00 [size=8]
	Region 1: I/O ports at fe00 [size=4]
	Region 2: I/O ports at fd00 [size=8]
	Region 3: I/O ports at fc00 [size=4]
	Region 4: I/O ports at fb00 [size=16]
	Region 5: Memory at fe02f000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [60] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: ahci
	Kernel modules: ahci

00:13.0 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI0) [1002:4387] (prog-if 10 [OHCI])
	Subsystem: ASUSTeK Computer Inc. Device [1043:81ef]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 17
	Region 0: Memory at fe02e000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd
	Kernel modules: ohci-hcd

00:13.1 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI1) [1002:4388] (prog-if 10 [OHCI])
	Subsystem: ASUSTeK Computer Inc. Device [1043:81ef]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 32 bytes
	Interrupt: pin B routed to IRQ 18
	Region 0: Memory at fe02d000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd
	Kernel modules: ohci-hcd

00:13.2 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI2) [1002:4389] (prog-if 10 [OHCI])
	Subsystem: ASUSTeK Computer Inc. Device [1043:81ef]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 32 bytes
	Interrupt: pin C routed to IRQ 19
	Region 0: Memory at fe02c000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd
	Kernel modules: ohci-hcd

00:13.3 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI3) [1002:438a] (prog-if 10 [OHCI])
	Subsystem: ASUSTeK Computer Inc. Device [1043:81ef]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 32 bytes
	Interrupt: pin B routed to IRQ 18
	Region 0: Memory at fe02b000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd
	Kernel modules: ohci-hcd

00:13.4 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI4) [1002:438b] (prog-if 10 [OHCI])
	Subsystem: ASUSTeK Computer Inc. Device [1043:81ef]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 32 bytes
	Interrupt: pin C routed to IRQ 19
	Region 0: Memory at fe02a000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd
	Kernel modules: ohci-hcd

00:13.5 USB Controller [0c03]: ATI Technologies Inc SB600 USB Controller (EHCI) [1002:4386] (prog-if 20 [EHCI])
	Subsystem: ASUSTeK Computer Inc. Device [1043:81ef]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 64 bytes
	Interrupt: pin D routed to IRQ 20
	Region 0: Memory at fe029000 (32-bit, non-prefetchable) [size=256]
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
		Bridge: PM- B3+
	Capabilities: [e4] Debug port: BAR=1 offset=00e0
	Kernel driver in use: ehci_hcd
	Kernel modules: ehci-hcd

00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller [1002:4385] (rev 14)
	Subsystem: ASUSTeK Computer Inc. Device [1043:81ef]
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Region 0: I/O ports at 0b00 [size=16]
	Capabilities: [b0] HyperTransport: MSI Mapping Enable- Fixed+

00:14.1 IDE interface [0101]: ATI Technologies Inc SB600 IDE [1002:438c] (prog-if 8a [Master SecP PriP])
	Subsystem: ASUSTeK Computer Inc. Device [1043:81ef]
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64
	Interrupt: pin A routed to IRQ 17
	Region 0: I/O ports at <unassigned>
	Region 1: I/O ports at <unassigned>
	Region 2: I/O ports at <unassigned>
	Region 3: I/O ports at <unassigned>
	Region 4: I/O ports at f900 [size=16]
	Kernel driver in use: ATIIXP_IDE
	Kernel modules: atiixp

00:14.3 ISA bridge [0601]: ATI Technologies Inc SB600 PCI to LPC Bridge [1002:438d]
	Subsystem: ASUSTeK Computer Inc. Device [1043:81ef]
	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI Bridge [1002:4384] (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=64
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: fde00000-fdefffff
	Prefetchable memory behind bridge: fdd00000-fddfffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration [1022:1100]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [80] HyperTransport: Host or Secondary Interface
		!!! Possibly incomplete decoding
		Command: WarmRst+ DblEnd-
		Link Control: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0
		Link Config: MLWI=16bit MLWO=16bit LWI=16bit LWO=16bit
		Revision ID: 1.02

00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map [1022:1101]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller [1022:1102]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control [1022:1103]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [f0] Secure device <?>

01:05.0 VGA compatible controller [0300]: ATI Technologies Inc RS690 [Radeon X1200 Series] [1002:791e] (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. Device [1043:826d]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 5
	Region 0: Memory at fa000000 (64-bit, prefetchable) [size=32M]
	Region 2: Memory at fdcf0000 (64-bit, non-prefetchable) [size=64K]
	Region 4: I/O ports at de00 [size=256]
	Region 5: Memory at fdb00000 (32-bit, non-prefetchable) [size=1M]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
		Address: 0000000000000000  Data: 0000

02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller [10ec:8168] (rev 01)
	Subsystem: ASUSTeK Computer Inc. Device [1043:81aa]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 20
	Region 0: I/O ports at ee00 [size=256]
	Region 2: Memory at fdaff000 (64-bit, non-prefetchable) [size=4K]
	[virtual] Expansion ROM at fdf00000 [disabled] [size=128K]
	Capabilities: [40] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] Vital Product Data <?>
	Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <1us, L1 unlimited
			ExtTag+ AttnBtn+ AttnInd+ PwrInd+ RBE- FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Suprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [84] Vendor Specific Information <?>
	Kernel driver in use: r8169
	Kernel modules: r8169


[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvops: AHCI problems with SB600

[Patrick Scharrenberg]

I forgot to mention that the version (2.6.30-rc3) from xen-tip/master
branch, which is checked out per default from the xen-unstable scripts,
does *not* have this issue and boots fine on that hardware.

Re: pvops: AHCI problems with SB600

[Konrad Rzeszutek Wilk]

On Sat, Sep 19, 2009 at 04:46:16PM +0200, Patrick Scharrenberg wrote:
> Hi,
> 
> I just tested this branch and found that the system gets stuck on
> initializing the AHCI controller (AMD SB600).

Ooooh. Can you give some more details on the hardware flavor? CPU? Memory?
What happens if you use dom0_mem=max:512MB or pass in 'irqpoll' to the
Linux kernel?

> Please find the error messages attached as png (unfortunately netconsole
> doesn't work either)

What are the errors for netconsole? Is it not initializing the right ethernet device?
Is the ethernet device compiled in? Can you paste what you have in your
grub config?

For example, this is what I have:

 kernel /xen.gz  com1=115200,8n1,0xcf00,0  console=com1,vga guest_loglvl=all  dom0_mem=max:512MB
 module /vmlinuz debug netconsole=6665@192.168.100.20/eth1,6666@192.168.100.16/00:25:11:18:A1:2E console=tty console=xvc0 irqpoll loglevel=10

That particular machine didn't come with a serial port. But the $14 PCI serial
card works without fault.

> 
> When booting the same kernel natively without xen, all works fine
> (including netconsole).

OK. I need the kernel boot log to figure out what might be a potential problem.
Let's try to get your netconsole working or your serial console? Does this
machine even have a serial port? Maybe you can use that to capture the kernel
output?

Re: pvops: AHCI problems with SB600

[Patrick Scharrenberg]

[-- Attachment #1: Type: text/plain, Size: 1148 bytes --]

Konrad Rzeszutek Wilk wrote:
> Ooooh. Can you give some more details on the hardware flavor? CPU? Memory?
Its an AMD system on an ASUS M2A-VM mainboard with an Athlon 64 X2 CPU
and 8GB of memory.

> What happens if you use dom0_mem=max:512MB or pass in 'irqpoll' to the
> Linux kernel?
irqpoll does not change anything but the memory limit does the trick.
The system then boots fine, I attached the dmesg log.
Since the netconsole now also works, I assume some kind of IRQ problem
which is strengthened by the fact, that the keyboard didn't work either
(for scrolling and searching for problems) without the memory limit option.

> OK. I need the kernel boot log to figure out what might be a potential problem.
> Let's try to get your netconsole working or your serial console? Does this
> machine even have a serial port? Maybe you can use that to capture the kernel
> output?

I attached the bootlog with the 512 memory-limit enabled and the xen dmesg.

Unfortunately I have to drive 200km to install a serial cable to the
system. So if there is an IRQ problem would the serial console really
help or would it suffer from the same irq problem?


[-- Attachment #2: linux-dmesg-2.6.31-maxmem512 --]
[-- Type: text/plain, Size: 65536 bytes --]

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.31 (root@brontus) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #2 SMP Sat Sep 19 15:53:55 CEST 2009
[    0.000000] Command line: root=/dev/mapper/brontus-xen_root ro debug netconsole=4444@1.2.3.4/eth0,6666@1.2.3.5/00:12:1E:19:D0:BC loglevel=10
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] xen_release_chunk: looking at area pfn 20000-ddee0
[    0.000000] xen_release_chunk: looking at area pfn ddf00-e0000
[    0.000000] xen_release_chunk: looking at area pfn f0000-fec00
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 000000000009f000 (usable)
[    0.000000]  Xen: 000000000009f000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000020000000 (usable)
[    0.000000]  Xen: 00000000ddee0000 - 00000000ddee3000 (ACPI NVS)
[    0.000000]  Xen: 00000000ddee3000 - 00000000ddef0000 (ACPI data)
[    0.000000]  Xen: 00000000ddef0000 - 00000000ddf00000 (reserved)
[    0.000000]  Xen: 00000000e0000000 - 00000000f0000000 (reserved)
[    0.000000]  Xen: 00000000fec00000 - 0000000100000000 (reserved)
[    0.000000] DMI 2.4 present.
[    0.000000] Phoenix BIOS detected: BIOS may corrupt low RAM, working around it.
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
[    0.000000] last_pfn = 0x20000 max_arch_pfn = 0x400000000
[    0.000000] Scanning 0 areas for low memory corruption
[    0.000000] modified physical RAM map:
[    0.000000]  modified: 0000000000000000 - 0000000000010000 (reserved)
[    0.000000]  modified: 0000000000010000 - 000000000009f000 (usable)
[    0.000000]  modified: 000000000009f000 - 0000000000100000 (reserved)
[    0.000000]  modified: 0000000000100000 - 0000000020000000 (usable)
[    0.000000]  modified: 00000000ddee0000 - 00000000ddee3000 (ACPI NVS)
[    0.000000]  modified: 00000000ddee3000 - 00000000ddef0000 (ACPI data)
[    0.000000]  modified: 00000000ddef0000 - 00000000ddf00000 (reserved)
[    0.000000]  modified: 00000000e0000000 - 00000000f0000000 (reserved)
[    0.000000]  modified: 00000000fec00000 - 0000000100000000 (reserved)
[    0.000000] initial memory mapped : 0 - 20000000
[    0.000000] init_memory_mapping: 0000000000000000-0000000020000000
[    0.000000]  0000000000 - 0020000000 page 4k
[    0.000000] kernel direct mapping tables up to 20000000 @ 100000-202000
[    0.000000] RAMDISK: 0199a000 - 02054200
[    0.000000] ACPI: RSDP 00000000000f8210 00024 (v02 ATI   )
[    0.000000] ACPI: XSDT 00000000ddee3100 00044 (v01 ATI    ASUSACPI 42302E31 AWRD 00000000)
[    0.000000] ACPI: FACP 00000000ddee8500 000F4 (v03 ATI    ASUSACPI 42302E31 AWRD 00000000)
[    0.000000] ACPI: DSDT 00000000ddee3280 05210 (v01 ATI    ASUSACPI 00001000 MSFT 03000000)
[    0.000000] ACPI: FACS 00000000ddee0000 00040
[    0.000000] ACPI: HPET 00000000ddee8740 00038 (v01 ATI    ASUSACPI 42302E31 AWRD 00000098)
[    0.000000] ACPI: MCFG 00000000ddee87c0 0003C (v01 ATI    ASUSACPI 42302E31 AWRD 00000000)
[    0.000000] ACPI: APIC 00000000ddee8640 00084 (v01 ATI    ASUSACPI 42302E31 AWRD 00000000)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] (8 early reservations) ==> bootmem [0000000000 - 0020000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
[    0.000000]   #1 [0002156000 - 000216b000]   XEN PAGETABLES ==> [0002156000 - 000216b000]
[    0.000000]   #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
[    0.000000]   #3 [0001000000 - 0001979274]    TEXT DATA BSS ==> [0001000000 - 0001979274]
[    0.000000]   #4 [000199a000 - 0002054200]          RAMDISK ==> [000199a000 - 0002054200]
[    0.000000]   #5 [0002055000 - 0002156000]   XEN START INFO ==> [0002055000 - 0002156000]
[    0.000000]   #6 [000197a000 - 000197a155]              BRK ==> [000197a000 - 000197a155]
[    0.000000]   #7 [0000100000 - 00001ea000]          PGTABLE ==> [0000100000 - 00001ea000]
[    0.000000] found SMP MP-table at [ffff8800000f6560] f6560
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00100000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x00020000
[    0.000000] On node 0 totalpages: 130959
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 237 pages reserved
[    0.000000]   DMA zone: 3690 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 1736 pages used for memmap
[    0.000000]   DMA32 zone: 125240 pages, LIFO batch:31
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] ACPI: PM-Timer IO Port: 0x4008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8200 base: 0xfed00000
[    0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 24
[    0.000000] PM: Registered nosave memory: 000000000009f000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 20000000 (gap: 20000000:bdee0000)
[    0.000000] NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Allocated 22 4k pages, static data 88032 bytes
[    0.000000] trying to map vcpu_info 0 at ffffc9000000b020, mfn 21617f, offset 32
[    0.000000] cpu 0 using vcpu_info at ffffc9000000b020
[    0.000000] trying to map vcpu_info 1 at ffffc90000023020, mfn 216195, offset 32
[    0.000000] cpu 1 using vcpu_info at ffffc90000023020
[    0.000000] Xen: using vcpu_info placement
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 128930
[    0.000000] Kernel command line: root=/dev/mapper/brontus-xen_root ro debug netconsole=4444@1.2.3.4/eth0,6666@1.2.3.5/00:12:1E:19:D0:BC loglevel=10
[    0.000000] PID hash table entries: 2048 (order: 11, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.000000] Initializing CPU#0
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ 20000000 size 32 MB
[    0.000000] Aperture too small (32 MB) than (64 MB)
[    0.000000] PCI-DMA: Using Xen software bounce buffering for IO (Xen-SWIOTLB)
[    0.000000] xen_swiotlb_fixup: buf=ffff880002c6c000 size=67108864
[    0.000000] xen_swiotlb_fixup: buf=ffff880006ccc000 size=32768
[    0.000000] Placing 64MB Xen software IO TLB between ffff880002c6c000 - ffff880006c6c000
[    0.000000] Xen software IO TLB at phys 0x2c6c000 - 0x6c6c000
[    0.000000] Memory: 430336k/524288k available (5097k kernel code, 452k absent, 92804k reserved, 2902k data, 568k init)
[    0.000000] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS:4352 nr_irqs:424
[    0.000000] xen: registering gsi 0 triggering 1 polarity 0
[    0.000000] xen: --> irq=0
[    0.000000] xen_set_ioapic_routing: irq 0 gsi 0 vector 0 ioapic 0 pin 0 triggering 0 polarity 0
[    0.000000] xen: registering gsi 1 triggering 1 polarity 0
[    0.000000] xen: --> irq=1
[    0.000000] xen_set_ioapic_routing: irq 1 gsi 1 vector 1 ioapic 0 pin 1 triggering 0 polarity 0
[    0.000000] xen: registering gsi 3 triggering 1 polarity 0
[    0.000000] xen: --> irq=3
[    0.000000] xen_set_ioapic_routing: irq 3 gsi 3 vector 3 ioapic 0 pin 3 triggering 0 polarity 0
[    0.000000] xen: registering gsi 4 triggering 1 polarity 0
[    0.000000] xen: --> irq=4
[    0.000000] xen_set_ioapic_routing: irq 4 gsi 4 vector 4 ioapic 0 pin 4 triggering 0 polarity 0
[    0.000000] xen: registering gsi 5 triggering 1 polarity 0
[    0.000000] xen: --> irq=5
[    0.000000] xen_set_ioapic_routing: irq 5 gsi 5 vector 5 ioapic 0 pin 5 triggering 0 polarity 0
[    0.000000] xen: registering gsi 6 triggering 1 polarity 0
[    0.000000] xen: --> irq=6
[    0.000000] xen_set_ioapic_routing: irq 6 gsi 6 vector 6 ioapic 0 pin 6 triggering 0 polarity 0
[    0.000000] xen: registering gsi 7 triggering 1 polarity 0
[    0.000000] xen: --> irq=7
[    0.000000] xen_set_ioapic_routing: irq 7 gsi 7 vector 7 ioapic 0 pin 7 triggering 0 polarity 0
[    0.000000] xen: registering gsi 8 triggering 1 polarity 0
[    0.000000] xen: --> irq=8
[    0.000000] xen_set_ioapic_routing: irq 8 gsi 8 vector 8 ioapic 0 pin 8 triggering 0 polarity 0
[    0.000000] xen: registering gsi 9 triggering 0 polarity 1
[    0.000000] xen: --> irq=9
[    0.000000] xen_set_ioapic_routing: irq 9 gsi 9 vector 9 ioapic 0 pin 9 triggering 1 polarity 1
[    0.000000] xen: registering gsi 10 triggering 1 polarity 0
[    0.000000] xen: --> irq=10
[    0.000000] xen_set_ioapic_routing: irq 10 gsi 10 vector 10 ioapic 0 pin 10 triggering 0 polarity 0
[    0.000000] xen: registering gsi 11 triggering 1 polarity 0
[    0.000000] xen: --> irq=11
[    0.000000] xen_set_ioapic_routing: irq 11 gsi 11 vector 11 ioapic 0 pin 11 triggering 0 polarity 0
[    0.000000] xen: registering gsi 12 triggering 1 polarity 0
[    0.000000] xen: --> irq=12
[    0.000000] xen_set_ioapic_routing: irq 12 gsi 12 vector 12 ioapic 0 pin 12 triggering 0 polarity 0
[    0.000000] xen: registering gsi 13 triggering 1 polarity 0
[    0.000000] xen: --> irq=13
[    0.000000] xen_set_ioapic_routing: irq 13 gsi 13 vector 13 ioapic 0 pin 13 triggering 0 polarity 0
[    0.000000] xen: registering gsi 14 triggering 1 polarity 0
[    0.000000] xen: --> irq=14
[    0.000000] xen_set_ioapic_routing: irq 14 gsi 14 vector 14 ioapic 0 pin 14 triggering 0 polarity 0
[    0.000000] xen: registering gsi 15 triggering 1 polarity 0
[    0.000000] xen: --> irq=15
[    0.000000] xen_set_ioapic_routing: irq 15 gsi 15 vector 15 ioapic 0 pin 15 triggering 0 polarity 0
[    0.000000] Detected 2399.914 MHz processor.
[    0.000999] Console: colour VGA+ 80x25
[    0.000999] console [tty0] enabled
[    0.000999] Xen: using vcpuop timer interface
[    0.000999] installing Xen timer for CPU 0
[    0.000999]   alloc irq_desc for 24 on node 0
[    0.000999]   alloc kstat_irqs on node 0
[    0.000999] Calibrating delay loop (skipped), value calculated using timer frequency.. 4799.82 BogoMIPS (lpj=2399914)
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] SELinux:  Starting in permissive mode
[    0.000999] Mount-cache hash table entries: 256
[    0.000999] Initializing cgroup subsys ns
[    0.000999] Initializing cgroup subsys cpuacct
[    0.000999] Initializing cgroup subsys freezer
[    0.000999] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.000999] CPU: L2 Cache: 512K (64 bytes/line)
[    0.001014] tseg: 00ddf00000
[    0.001080] CPU: Physical Processor ID: 0
[    0.001147] CPU: Processor Core ID: 0
[    0.001218] mce: CPU supports 5 MCE banks
[    0.001313] Performance Counters: AMD PMU driver.
[    0.001408] ------------[ cut here ]------------
[    0.001480] WARNING: at /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/arch/x86/xen/enlighten.c:714 xen_apic_write+0x15/0x17()
[    0.001586] Hardware name: System Product Name
[    0.001652] Modules linked in:
[    0.001745] Pid: 0, comm: swapper Not tainted 2.6.31 #2
[    0.001812] Call Trace:
[    0.001882]  [<ffffffff8100b361>] ? xen_apic_write+0x15/0x17
[    0.001953]  [<ffffffff8104d22b>] warn_slowpath_common+0x77/0xa4
[    0.001999]  [<ffffffff8104d267>] warn_slowpath_null+0xf/0x11
[    0.001999]  [<ffffffff8100b361>] xen_apic_write+0x15/0x17
[    0.001999]  [<ffffffff8101f09b>] perf_counters_lapic_init+0x2e/0x30
[    0.001999]  [<ffffffff817f113c>] init_hw_perf_counters+0x305/0x3a2
[    0.001999]  [<ffffffff8100f07f>] ? xen_restore_fl_direct_end+0x0/0x1
[    0.001999]  [<ffffffff810dc633>] ? kmem_cache_alloc+0x7b/0xdd
[    0.001999]  [<ffffffff817f0e08>] identify_boot_cpu+0x3c/0x3e
[    0.001999]  [<ffffffff817f0e13>] check_bugs+0x9/0x2d
[    0.001999]  [<ffffffff817e856a>] start_kernel+0x3d7/0x3ec
[    0.001999]  [<ffffffff817e7a9f>] x86_64_start_reservations+0xaa/0xae
[    0.001999]  [<ffffffff817eb8c9>] xen_start_kernel+0x61e/0x625
[    0.001999] ---[ end trace 4eaa2a86a8e2da22 ]---
[    0.001999] ... version:                 0
[    0.001999] ... bit width:               48
[    0.001999] ... generic counters:        4
[    0.001999] ... value mask:              0000ffffffffffff
[    0.001999] ... max period:              00007fffffffffff
[    0.001999] ... fixed-purpose counters:  0
[    0.001999] ... counter mask:            000000000000000f
[    0.001999] SMP alternatives: switching to UP code
[    0.002683] ACPI: Core revision 20090521
[    0.011105]   alloc irq_desc for 25 on node 0
[    0.011176]   alloc kstat_irqs on node 0
[    0.011250] cpu 0 spinlock event irq 25
[    0.011319]   alloc irq_desc for 26 on node 0
[    0.011385]   alloc kstat_irqs on node 0
[    0.011455]   alloc irq_desc for 27 on node 0
[    0.011522]   alloc kstat_irqs on node 0
[    0.011592]   alloc irq_desc for 28 on node 0
[    0.011659]   alloc kstat_irqs on node 0
[    0.011731]   alloc irq_desc for 29 on node 0
[    0.011798]   alloc kstat_irqs on node 0
[    0.012061] installing Xen timer for CPU 1
[    0.012130]   alloc irq_desc for 30 on node 0
[    0.012196]   alloc kstat_irqs on node 0
[    0.012268]   alloc irq_desc for 31 on node 0
[    0.012335]   alloc kstat_irqs on node 0
[    0.012406] cpu 1 spinlock event irq 31
[    0.012502] SMP alternatives: switching to SMP code
[    0.012998]   alloc irq_desc for 32 on node 0
[    0.012998]   alloc kstat_irqs on node 0
[    0.013015]   alloc irq_desc for 33 on node 0
[    0.013081]   alloc kstat_irqs on node 0
[    0.013152]   alloc irq_desc for 34 on node 0
[    0.013218]   alloc kstat_irqs on node 0
[    0.013287]   alloc irq_desc for 35 on node 0
[    0.013353]   alloc kstat_irqs on node 0
[    0.000018] Initializing CPU#1
[    0.000082] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[    0.000086] CPU: L2 Cache: 512K (64 bytes/line)
[    0.000090] CPU: Physical Processor ID: 0
[    0.000092] CPU: Processor Core ID: 1
[    0.000100] mce: CPU supports 5 MCE banks
[    0.013592] Brought up 2 CPUs
[    0.014692] Booting paravirtualized kernel on Xen
[    0.014692] Xen version: 3.5-unstable (preserve-AD) (dom0)
[    0.015050] Grant table initialized
[    0.015150] Time:  8:51:59  Date: 09/22/09
[    0.015301] NET: Registered protocol family 16
[    0.015420]   alloc irq_desc for 36 on node 0
[    0.015420]   alloc kstat_irqs on node 0
[    0.016038] xenbus_probe_init ok
[    0.016299] node 0 link 0: io port [c000, ffff]
[    0.016372] TOM: 00000000e0000000 aka 3584M
[    0.016440] node 0 link 0: mmio [a0000, bffff]
[    0.016537] node 0 link 0: mmio [e0000000, dfffffff]
[    0.016633] node 0 link 0: mmio [f0000000, fe02ffff]
[    0.016728] node 0 link 0: mmio [e0000000, e03fffff]
[    0.017022] TOM2: 0000000220000000 aka 8704M
[    0.017089] bus: [00,03] on node 0 link 0
[    0.017156] bus: 00 index 0 io port: [0, ffff]
[    0.017222] bus: 00 index 1 mmio: [a0000, bffff]
[    0.017289] bus: 00 index 2 mmio: [e0000000, efffffff]
[    0.017356] bus: 00 index 3 mmio: [f0000000, ffffffff]
[    0.017423] bus: 00 index 4 mmio: [220000000, fcffffffff]
[    0.017513] ACPI: bus type pci registered
[    0.017615] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
[    0.017615] PCI: MCFG area at e0000000 reserved in E820
[    0.086288] PCI: Using MMCONFIG at e0000000 - efffffff
[    0.086363] PCI: Using configuration type 1 for base access
[    0.100119] bio: create slab <bio-0> at 0
[    0.102149] ACPI: EC: Look up EC in DSDT
[    0.114184] ACPI: Interpreter enabled
[    0.114254] ACPI: (supports S0 S1 S3 S4 S5)
[    0.114501] ACPI: Using IOAPIC for interrupt routing
[    0.125358] ACPI: No dock devices found.
[    0.125606] ACPI: PCI Root Bridge [PCI0] (0000:00)
[    0.126089] pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
[    0.126160] pci 0000:00:07.0: PME# disabled
[    0.126336] pci 0000:00:12.0: reg 10 io port: [0xff00-0xff07]
[    0.126413] pci 0000:00:12.0: reg 14 io port: [0xfe00-0xfe03]
[    0.126490] pci 0000:00:12.0: reg 18 io port: [0xfd00-0xfd07]
[    0.126568] pci 0000:00:12.0: reg 1c io port: [0xfc00-0xfc03]
[    0.126645] pci 0000:00:12.0: reg 20 io port: [0xfb00-0xfb0f]
[    0.126723] pci 0000:00:12.0: reg 24 32bit mmio: [0xfe02f000-0xfe02f3ff]
[    0.126886] pci 0000:00:13.0: reg 10 32bit mmio: [0xfe02e000-0xfe02efff]
[    0.127021] pci 0000:00:13.1: reg 10 32bit mmio: [0xfe02d000-0xfe02dfff]
[    0.127199] pci 0000:00:13.2: reg 10 32bit mmio: [0xfe02c000-0xfe02cfff]
[    0.127378] pci 0000:00:13.3: reg 10 32bit mmio: [0xfe02b000-0xfe02bfff]
[    0.127557] pci 0000:00:13.4: reg 10 32bit mmio: [0xfe02a000-0xfe02afff]
[    0.127772] pci 0000:00:13.5: reg 10 32bit mmio: [0xfe029000-0xfe0290ff]
[    0.127945] pci 0000:00:13.5: supports D1 D2
[    0.127980] pci 0000:00:13.5: PME# supported from D0 D1 D2 D3hot
[    0.127980] pci 0000:00:13.5: PME# disabled
[    0.128057] pci 0000:00:14.0: reg 10 io port: [0xb00-0xb0f]
[    0.128262] pci 0000:00:14.1: reg 10 io port: [0x00-0x07]
[    0.128340] pci 0000:00:14.1: reg 14 io port: [0x00-0x03]
[    0.128416] pci 0000:00:14.1: reg 18 io port: [0x00-0x07]
[    0.128493] pci 0000:00:14.1: reg 1c io port: [0x00-0x03]
[    0.128569] pci 0000:00:14.1: reg 20 io port: [0xf900-0xf90f]
[    0.129249] pci 0000:01:05.0: reg 10 64bit mmio: [0xfa000000-0xfbffffff]
[    0.129335] pci 0000:01:05.0: reg 18 64bit mmio: [0xfdbf0000-0xfdbfffff]
[    0.129412] pci 0000:01:05.0: reg 20 io port: [0xde00-0xdeff]
[    0.129488] pci 0000:01:05.0: reg 24 32bit mmio: [0xfda00000-0xfdafffff]
[    0.129595] pci 0000:01:05.0: supports D1 D2
[    0.129739] pci 0000:00:01.0: bridge io port: [0xd000-0xdfff]
[    0.129813] pci 0000:00:01.0: bridge 32bit mmio: [0xfda00000-0xfdbfffff]
[    0.129891] pci 0000:00:01.0: bridge 64bit mmio pref: [0xfa000000-0xfbffffff]
[    0.129980] pci 0000:02:00.0: reg 10 io port: [0xee00-0xeeff]
[    0.129980] pci 0000:02:00.0: reg 18 64bit mmio: [0xfdfff000-0xfdffffff]
[    0.129980] pci 0000:02:00.0: reg 30 32bit mmio: [0x000000-0x01ffff]
[    0.129980] pci 0000:02:00.0: supports D1 D2
[    0.129980] pci 0000:02:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.129980] pci 0000:02:00.0: PME# disabled
[    0.130088] pci 0000:00:07.0: bridge io port: [0xe000-0xefff]
[    0.130159] pci 0000:00:07.0: bridge 32bit mmio: [0xfdf00000-0xfdffffff]
[    0.130233] pci 0000:00:07.0: bridge 64bit mmio pref: [0xfdc00000-0xfdcfffff]
[    0.130423] pci 0000:00:14.4: transparent bridge
[    0.130495] pci 0000:00:14.4: bridge io port: [0xc000-0xcfff]
[    0.130567] pci 0000:00:14.4: bridge 32bit mmio: [0xfde00000-0xfdefffff]
[    0.130639] pci 0000:00:14.4: bridge 32bit mmio pref: [0xfdd00000-0xfddfffff]
[    0.130733] pci_bus 0000:00: on NUMA node 0
[    0.130804] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    0.131395] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P2P_._PRT]
[    0.131745] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE7._PRT]
[    0.131914] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGP_._PRT]
[    0.173113] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 11) *0, disabled.
[    0.173657] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10 11) *0, disabled.
[    0.174103] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 10 11) *0, disabled.
[    0.175102] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 11) *0, disabled.
[    0.175644] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11) *0, disabled.
[    0.176102] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11) *0, disabled.
[    0.176643] ACPI: PCI Interrupt Link [LNK0] (IRQs 3 4 5 6 7 10 *11)
[    0.177073] ACPI: PCI Interrupt Link [LNK1] (IRQs 3 4 5 6 7 10 11) *0, disabled.
[    0.177572] xenbus_probe_backend_init bus registered ok
[    0.177572] xenbus_probe_frontend_init bus registered ok
[    0.177572] xen_balloon: Initialising balloon driver.
[    0.178140] SCSI subsystem initialized
[    0.178140] libata version 3.00 loaded.
[    0.179053] usbcore: registered new interface driver usbfs
[    0.179116] usbcore: registered new interface driver hub
[    0.179116] usbcore: registered new device driver usb
[    0.180182] PCI: Using ACPI for IRQ routing
[    0.180446] IO APIC resources couldn't be allocated.
[    0.194046] cfg80211: Using static regulatory domain info
[    0.194058] cfg80211: Regulatory domain: US
[    0.194124] 	(start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
[    0.194232] 	(2402000 KHz - 2472000 KHz @ 40000 KHz), (600 mBi, 2700 mBm)
[    0.194305] 	(5170000 KHz - 5190000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
[    0.194376] 	(5190000 KHz - 5210000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
[    0.194446] 	(5210000 KHz - 5230000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
[    0.194516] 	(5230000 KHz - 5330000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
[    0.194586] 	(5735000 KHz - 5835000 KHz @ 40000 KHz), (600 mBi, 3000 mBm)
[    0.194667] cfg80211: Calling CRDA for country: US
[    0.194772] NetLabel: Initializing
[    0.194772] NetLabel:  domain hash size = 128
[    0.194772] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.194772] NetLabel:  unlabeled traffic allowed by default
[    0.195983] Switched to high resolution mode on CPU 0
[    0.196210] Switched to high resolution mode on CPU 1
[    0.202027] pnp: PnP ACPI init
[    0.202119] ACPI: bus type pnp registered
[    0.203617] xen: registering gsi 8 triggering 1 polarity 0
[    0.203686] xen_allocate_pirq: returning irq 8 for gsi 8
[    0.203752] xen: --> irq=8
[    0.203819] xen_set_ioapic_routing: irq 8 gsi 8 vector 8 ioapic 0 pin 8 triggering 0 polarity 0
[    0.204403] xen: registering gsi 13 triggering 1 polarity 0
[    0.204472] xen_allocate_pirq: returning irq 13 for gsi 13
[    0.204539] xen: --> irq=13
[    0.204605] xen_set_ioapic_routing: irq 13 gsi 13 vector 13 ioapic 0 pin 13 triggering 0 polarity 0
[    0.207199] pnp: PnP ACPI: found 10 devices
[    0.207268] ACPI: ACPI bus type pnp unregistered
[    0.207351] system 00:01: ioport range 0x4100-0x411f has been reserved
[    0.207419] system 00:01: ioport range 0x228-0x22f has been reserved
[    0.207486] system 00:01: ioport range 0x40b-0x40b has been reserved
[    0.207553] system 00:01: ioport range 0x4d6-0x4d6 has been reserved
[    0.207621] system 00:01: ioport range 0xc00-0xc01 has been reserved
[    0.207688] system 00:01: ioport range 0xc14-0xc14 has been reserved
[    0.207755] system 00:01: ioport range 0xc50-0xc52 has been reserved
[    0.207823] system 00:01: ioport range 0xc6c-0xc6d has been reserved
[    0.207890] system 00:01: ioport range 0xc6f-0xc6f has been reserved
[    0.207958] system 00:01: ioport range 0xcd0-0xcd1 has been reserved
[    0.208026] system 00:01: ioport range 0xcd2-0xcd3 has been reserved
[    0.208093] system 00:01: ioport range 0xcd4-0xcdf has been reserved
[    0.208160] system 00:01: ioport range 0x4000-0x40fe has been reserved
[    0.208172] system 00:01: ioport range 0x4210-0x4217 has been reserved
[    0.208172] system 00:01: ioport range 0xb10-0xb1f has been reserved
[    0.208172] system 00:07: ioport range 0x4d0-0x4d1 has been reserved
[    0.208172] system 00:07: ioport range 0x220-0x225 has been reserved
[    0.208172] system 00:08: iomem range 0xe0000000-0xefffffff has been reserved
[    0.208172] system 00:09: iomem range 0xcd600-0xcffff could not be reserved
[    0.208172] system 00:09: iomem range 0xf0000-0xf7fff could not be reserved
[    0.208172] system 00:09: iomem range 0xf8000-0xfbfff could not be reserved
[    0.208172] system 00:09: iomem range 0xfc000-0xfffff could not be reserved
[    0.208172] system 00:09: iomem range 0xddef0000-0xddfeffff could not be reserved
[    0.208172] system 00:09: iomem range 0xfed00000-0xfed000ff has been reserved
[    0.208172] system 00:09: iomem range 0xddee0000-0xddefffff could not be reserved
[    0.208172] system 00:09: iomem range 0xffff0000-0xffffffff has been reserved
[    0.208172] system 00:09: iomem range 0x0-0x9ffff could not be reserved
[    0.208172] system 00:09: iomem range 0x100000-0xddedffff could not be reserved
[    0.208172] system 00:09: iomem range 0xddff0000-0xdffeffff has been reserved
[    0.208172] system 00:09: iomem range 0xfec00000-0xfec00fff has been reserved
[    0.208172] system 00:09: iomem range 0xfee00000-0xfee00fff has been reserved
[    0.208172] system 00:09: iomem range 0xfff80000-0xfffeffff has been reserved
[    0.215470] pci 0000:00:01.0: PCI bridge, secondary bus 0000:01
[    0.215541] pci 0000:00:01.0:   IO window: 0xd000-0xdfff
[    0.215612] pci 0000:00:01.0:   MEM window: 0xfda00000-0xfdbfffff
[    0.215683] pci 0000:00:01.0:   PREFETCH window: 0x000000fa000000-0x000000fbffffff
[    0.215797] pci 0000:00:07.0: PCI bridge, secondary bus 0000:02
[    0.215866] pci 0000:00:07.0:   IO window: 0xe000-0xefff
[    0.215937] pci 0000:00:07.0:   MEM window: 0xfdf00000-0xfdffffff
[    0.216002] pci 0000:00:07.0:   PREFETCH window: 0x000000fdc00000-0x000000fdcfffff
[    0.216002] pci 0000:00:14.4: PCI bridge, secondary bus 0000:03
[    0.216002] pci 0000:00:14.4:   IO window: 0xc000-0xcfff
[    0.216002] pci 0000:00:14.4:   MEM window: 0xfde00000-0xfdefffff
[    0.216002] pci 0000:00:14.4:   PREFETCH window: 0xfdd00000-0xfddfffff
[    0.216002] pci 0000:00:07.0: setting latency timer to 64
[    0.216002] pci_bus 0000:00: resource 0 io:  [0x00-0xffff]
[    0.216002] pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffffffffffff]
[    0.216002] pci_bus 0000:01: resource 0 io:  [0xd000-0xdfff]
[    0.216002] pci_bus 0000:01: resource 1 mem: [0xfda00000-0xfdbfffff]
[    0.216002] pci_bus 0000:01: resource 2 pref mem [0xfa000000-0xfbffffff]
[    0.216002] pci_bus 0000:02: resource 0 io:  [0xe000-0xefff]
[    0.216002] pci_bus 0000:02: resource 1 mem: [0xfdf00000-0xfdffffff]
[    0.216002] pci_bus 0000:02: resource 2 pref mem [0xfdc00000-0xfdcfffff]
[    0.216002] pci_bus 0000:03: resource 0 io:  [0xc000-0xcfff]
[    0.216002] pci_bus 0000:03: resource 1 mem: [0xfde00000-0xfdefffff]
[    0.216002] pci_bus 0000:03: resource 2 pref mem [0xfdd00000-0xfddfffff]
[    0.216002] pci_bus 0000:03: resource 3 io:  [0x00-0xffff]
[    0.216002] pci_bus 0000:03: resource 4 mem: [0x000000-0xffffffffffffffff]
[    0.217435] NET: Registered protocol family 2
[    0.217636] IP route cache hash table entries: 16384 (order: 5, 131072 bytes)
[    0.218227] TCP established hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.218758] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.219365] TCP: Hash tables configured (established 65536 bind 65536)
[    0.219439] TCP reno registered
[    0.219662] NET: Registered protocol family 1
[    0.219824] Trying to unpack rootfs image as initramfs...
[    0.230681] Freeing initrd memory: 6888k freed
[    0.237071] Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    0.237185] Scanning for low memory corruption every 60 seconds
[    0.237776] audit: initializing netlink socket (disabled)
[    0.237864] type=2000 audit(1253609519.993:1): initialized
[    0.245445] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.251181] VFS: Disk quotas dquot_6.5.2
[    0.251435] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.254044] msgmni has been set to 855
[    0.254411] SELinux:  Registering netfilter hooks
[    0.255264] alg: No test for stdrng (krng)
[    0.255568] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    0.255674] io scheduler noop registered
[    0.255740] io scheduler anticipatory registered
[    0.255807] io scheduler deadline registered
[    0.255886] io scheduler cfq registered (default)
[    0.472067] pci 0000:01:05.0: Boot video device
[    0.472476]   alloc irq_desc for 37 on node 0
[    0.472543]   alloc kstat_irqs on node 0
[    0.472639] pcieport-driver 0000:00:07.0: setting latency timer to 64
[    0.473127] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.473758] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    0.473866] ACPI: Power Button [PWRF]
[    0.474094] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input1
[    0.474200] ACPI: Power Button [PWRB]
[    0.474490] fan PNP0C0B:00: registered as cooling_device0
[    0.474566] ACPI: Fan [FAN] (on)
[    0.475022] processor LNXCPU:00: registered as cooling_device1
[    0.475243] processor LNXCPU:01: registered as cooling_device2
[    0.475451] processor LNXCPU:02: registered as cooling_device3
[    0.475658] processor LNXCPU:03: registered as cooling_device4
[    0.479944] ACPI Warning: \_TZ_.THRM._PSL: Return Package type mismatch at index 0 - found [NULL Object Descriptor], expected Reference 20090521 nspredef-946
[    0.480003] ACPI: Expecting a [Reference] package element, found type 0
[    0.480003] ACPI: Invalid passive threshold
[    0.480721] thermal LNXTHERM:01: registered as thermal_zone0
[    0.480809] ACPI: Thermal Zone [THRM] (40 C)
[    0.481131] Event-channel device installed.
[    0.485298] registering netback
[    0.492220]   alloc irq_desc for 38 on node 0
[    0.492293]   alloc kstat_irqs on node 0
[    0.493427] hpet_acpi_add: no address or irqs in _CRS
[    0.493686] Non-volatile memory driver v1.3
[    0.493955] Linux agpgart interface v0.103
[    0.494281] [drm] Initialized drm 1.1.0 20060810
[    0.494508] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.499373] brd: module loaded
[    0.501396] loop: module loaded
[    0.501689] input: Macintosh mouse button emulation as /devices/virtual/input/input2
[    0.502407] ahci 0000:00:12.0: version 3.0
[    0.502495] xen: registering gsi 22 triggering 0 polarity 1
[    0.502565]   alloc irq_desc for 22 on node 0
[    0.503349]   alloc kstat_irqs on node 0
[    0.505513] xen: --> irq=22
[    0.505580] xen_set_ioapic_routing: irq 22 gsi 22 vector 22 ioapic 0 pin 22 triggering 1 polarity 1
[    0.505696] ahci 0000:00:12.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
[    0.505885] ahci 0000:00:12.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0xf impl SATA mode
[    0.505992] ahci 0000:00:12.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part 
[    0.506973] scsi0 : ahci
[    0.507523] scsi1 : ahci
[    0.507822] scsi2 : ahci
[    0.508132] scsi3 : ahci
[    0.508634] ata1: SATA max UDMA/133 abar m1024@0xfe02f000 port 0xfe02f100 irq 22
[    0.508745] ata2: SATA max UDMA/133 abar m1024@0xfe02f000 port 0xfe02f180 irq 22
[    0.508856] ata3: SATA max UDMA/133 abar m1024@0xfe02f000 port 0xfe02f200 irq 22
[    0.508965] ata4: SATA max UDMA/133 abar m1024@0xfe02f000 port 0xfe02f280 irq 22
[    0.509326] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    0.509421] xen: registering gsi 19 triggering 0 polarity 1
[    0.509495]   alloc irq_desc for 19 on node 0
[    0.509564]   alloc kstat_irqs on node 0
[    0.509635] xen: --> irq=19
[    0.509704] xen_set_ioapic_routing: irq 19 gsi 19 vector 19 ioapic 0 pin 19 triggering 1 polarity 1
[    0.509824] r8169 0000:02:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
[    0.509964] r8169 0000:02:00.0: setting latency timer to 64
[    0.510100]   alloc irq_desc for 39 on node 0
[    0.510100]   alloc kstat_irqs on node 0
[    0.510834] eth0: RTL8168b/8111b at 0xffffc9000007e000, 00:1b:fc:86:73:27, XID 38000000 IRQ 39
[    0.512532] netconsole: local port 4444
[    0.512600] netconsole: local IP 1.2.3.4
[    0.512666] netconsole: interface eth0
[    0.512732] netconsole: remote port 6666
[    0.512798] netconsole: remote IP 1.2.3.5
[    0.512864] netconsole: remote ethernet address 00:12:1e:19:d0:bc
[    0.512932] netconsole: device eth0 not up yet, forcing it
[    0.513449] r8169: eth0: link down
[    0.513637] r8169: eth0: link down
[    0.814306] ata2: SATA link down (SStatus 0 SControl 300)
[    0.966026] ata1: softreset failed (device not ready)
[    0.966095] ata1: applying SB600 PMP SRST workaround and retrying
[    0.967268] ata3: softreset failed (device not ready)
[    0.967336] ata3: applying SB600 PMP SRST workaround and retrying
[    0.968269] ata4: softreset failed (device not ready)
[    0.968337] ata4: applying SB600 PMP SRST workaround and retrying
[    1.119035] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[    1.120278] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[    1.121280] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.148058] ata1.00: ATA-8: ST3500320NS, SN06, max UDMA/133
[    1.148129] ata1.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32)
[    1.148211] ata1.00: SB600 AHCI: limiting to 255 sectors per cmd
[    1.149321] ata3.00: ATA-8: ST3500320NS, SN06, max UDMA/133
[    1.149389] ata3.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32)
[    1.149470] ata3.00: SB600 AHCI: limiting to 255 sectors per cmd
[    1.150260] ata4.00: ATA-8: ST3500320NS, SN06, max UDMA/133
[    1.150327] ata4.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32)
[    1.150408] ata4.00: SB600 AHCI: limiting to 255 sectors per cmd
[    1.190547] ata1.00: SB600 AHCI: limiting to 255 sectors per cmd
[    1.190616] ata1.00: configured for UDMA/133
[    1.191801] ata3.00: SB600 AHCI: limiting to 255 sectors per cmd
[    1.191870] ata3.00: configured for UDMA/133
[    1.192677] ata4.00: SB600 AHCI: limiting to 255 sectors per cmd
[    1.192745] ata4.00: configured for UDMA/133
[    1.201164] scsi 0:0:0:0: Direct-Access     ATA      ST3500320NS      SN06 PQ: 0 ANSI: 5
[    1.201799] sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB)
[    1.201866] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    1.202146] scsi 2:0:0:0: Direct-Access     ATA      ST3500320NS      SN06 PQ: 0 ANSI: 5
[    1.202149] sd 0:0:0:0: [sda] Write Protect is off
[    1.202155] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    1.202211] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.202485]  sda:
[    1.202913] sd 2:0:0:0: [sdb] 976773168 512-byte logical blocks: (500 GB/465 GiB)
[    1.203185] sd 2:0:0:0: [sdb] Write Protect is off
[    1.203255] sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    1.203382] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.203755]  sdb:
[    1.203994] sd 2:0:0:0: Attached scsi generic sg1 type 0
[    1.204355] scsi 3:0:0:0: Direct-Access     ATA      ST3500320NS      SN06 PQ: 0 ANSI: 5
[    1.204879] sd 3:0:0:0: [sdc] 976773168 512-byte logical blocks: (500 GB/465 GiB)
[    1.205091] sd 3:0:0:0: [sdc] Write Protect is off
[    1.205159] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    1.205313] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.205660]  sdc:
[    1.205918] sd 3:0:0:0: Attached scsi generic sg2 type 0
[    1.216527]  sda1 sda4
[    1.217357] sd 0:0:0:0: [sda] Attached SCSI disk
[    1.219995]  sdb1 sdb4
[    1.220772] sd 2:0:0:0: [sdb] Attached SCSI disk
[    1.223383]  sdc1 sdc4
[    1.224200] sd 3:0:0:0: [sdc] Attached SCSI disk
[    2.147642] r8169: eth0: link up
[    2.153019] console [netcon0] enabled
[    2.154004] netconsole: network logging started
[    2.207689] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.207767] ehci_hcd: block sizes: qh 192 qtd 96 itd 192 sitd 96
[    2.207891] xen: registering gsi 19 triggering 0 polarity 1
[    2.207963] xen_allocate_pirq: returning irq 19 for gsi 19
[    2.208035] xen: --> irq=19
[    2.208106] xen_set_ioapic_routing: irq 19 gsi 19 vector 19 ioapic 0 pin 19 triggering 1 polarity 1
[    2.208227] ehci_hcd 0000:00:13.5: PCI INT D -> GSI 19 (level, low) -> IRQ 19
[    2.208328] ehci_hcd 0000:00:13.5: EHCI Host Controller
[    2.208454] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file 'devices'
[    2.208580] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '001'
[    2.208817] ehci_hcd 0000:00:13.5: new USB bus registered, assigned bus number 1
[    2.208945] ehci_hcd 0000:00:13.5: reset hcs_params 0x10520a dbg=1 cc=5 pcc=2 ordered !ppc ports=10
[    2.209059] ehci_hcd 0000:00:13.5: reset hcc_params a072 thresh 7 uframes 256/512/1024
[    2.209240] ehci_hcd 0000:00:13.5: applying AMD SB600/SB700 USB freeze workaround
[    2.209366] ehci_hcd 0000:00:13.5: reset command 080002 (park)=0 ithresh=8 period=1024 Reset HALT
[    2.209497] ehci_hcd 0000:00:13.5: debug port 1
[    2.209581] ehci_hcd 0000:00:13.5: MWI active
[    2.209652] ehci_hcd 0000:00:13.5: supports USB remote wakeup
[    2.209754] ehci_hcd 0000:00:13.5: irq 19, io mem 0xfe029000
[    2.209805] ehci_hcd 0000:00:13.5: reset command 080002 (park)=0 ithresh=8 period=1024 Reset HALT
[    2.209805] ehci_hcd 0000:00:13.5: init command 010009 (park)=0 ithresh=1 period=256 RUN
[    2.216018] ehci_hcd 0000:00:13.5: USB 2.0 started, EHCI 1.00
[    2.216162] usb usb1: default language 0x0409
[    2.216253] usb usb1: udev 1, busnum 1, minor = 0
[    2.216324] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    2.216395] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.216501] usb usb1: Product: EHCI Host Controller
[    2.216571] usb usb1: Manufacturer: Linux 2.6.31 ehci_hcd
[    2.216640] usb usb1: SerialNumber: 0000:00:13.5
[    2.216781] usb usb1: uevent
[    2.216959] usb usb1: usb_probe_device
[    2.217032] usb usb1: configuration #1 chosen from 1 choice
[    2.217131] usb usb1: adding 1-0:1.0 (config #1, interface 0)
[    2.217233] usb 1-0:1.0: uevent
[    2.217394] hub 1-0:1.0: usb_probe_interface
[    2.217465] hub 1-0:1.0: usb_probe_interface - got id
[    2.217536] hub 1-0:1.0: USB hub found
[    2.217641] hub 1-0:1.0: 10 ports detected
[    2.217712] hub 1-0:1.0: standalone hub
[    2.217781] hub 1-0:1.0: no power switching (usb 1.0)
[    2.217850] hub 1-0:1.0: individual port over-current protection
[    2.217920] hub 1-0:1.0: power on to power good time: 20ms
[    2.218018] hub 1-0:1.0: local power source is good
[    2.218090] hub 1-0:1.0: trying to enable port power on non-switchable hub
[    2.218245] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '001'
[    2.218492] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.218563] ohci_hcd: block sizes: ed 80 td 96
[    2.218671] xen: registering gsi 16 triggering 0 polarity 1
[    2.218745]   alloc irq_desc for 16 on node 0
[    2.218815]   alloc kstat_irqs on node 0
[    2.218894] xen: --> irq=16
[    2.218964] xen_set_ioapic_routing: irq 16 gsi 16 vector 16 ioapic 0 pin 16 triggering 1 polarity 1
[    2.219081] ohci_hcd 0000:00:13.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    2.219170] ohci_hcd 0000:00:13.0: OHCI Host Controller
[    2.219276] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '002'
[    2.219512] ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 2
[    2.219666] ohci_hcd 0000:00:13.0: created debug files
[    2.219739] ohci_hcd 0000:00:13.0: supports USB remote wakeup
[    2.219836] ohci_hcd 0000:00:13.0: irq 16, io mem 0xfe02e000
[    2.274046] ohci_hcd 0000:00:13.0: OHCI controller state
[    2.274119] ohci_hcd 0000:00:13.0: OHCI 1.0, NO legacy support registers
[    2.274193] ohci_hcd 0000:00:13.0: control 0x283 RWC HCFS=operational CBSR=3
[    2.274265] ohci_hcd 0000:00:13.0: cmdstatus 0x00000 SOC=0
[    2.274336] ohci_hcd 0000:00:13.0: intrstatus 0x00000004 SF
[    2.274408] ohci_hcd 0000:00:13.0: intrenable 0x8000001a MIE UE RD WDH
[    2.274486] ohci_hcd 0000:00:13.0: hcca frame #0005
[    2.274557] ohci_hcd 0000:00:13.0: roothub.a 02000202 POTPGT=2 NPS NDP=2(2)
[    2.274628] ohci_hcd 0000:00:13.0: roothub.b 00000000 PPCM=0000 DR=0000
[    2.274700] ohci_hcd 0000:00:13.0: roothub.status 00008000 DRWE
[    2.274772] ohci_hcd 0000:00:13.0: roothub.portstatus [0] 0x00000100 PPS
[    2.274845] ohci_hcd 0000:00:13.0: roothub.portstatus [1] 0x00000100 PPS
[    2.274952] usb usb2: default language 0x0409
[    2.275050] usb usb2: udev 1, busnum 2, minor = 128
[    2.275121] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[    2.275192] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.275300] usb usb2: Product: OHCI Host Controller
[    2.275370] usb usb2: Manufacturer: Linux 2.6.31 ohci_hcd
[    2.275439] usb usb2: SerialNumber: 0000:00:13.0
[    2.275575] usb usb2: uevent
[    2.275739] usb usb2: usb_probe_device
[    2.275810] usb usb2: configuration #1 chosen from 1 choice
[    2.275896] usb usb2: adding 2-0:1.0 (config #1, interface 0)
[    2.275995] usb 2-0:1.0: uevent
[    2.276159] hub 2-0:1.0: usb_probe_interface
[    2.276230] hub 2-0:1.0: usb_probe_interface - got id
[    2.276300] hub 2-0:1.0: USB hub found
[    2.276408] hub 2-0:1.0: 2 ports detected
[    2.276479] hub 2-0:1.0: standalone hub
[    2.276548] hub 2-0:1.0: no power switching (usb 1.0)
[    2.276617] hub 2-0:1.0: global over-current protection
[    2.276686] hub 2-0:1.0: power on to power good time: 4ms
[    2.276775] hub 2-0:1.0: local power source is good
[    2.276844] hub 2-0:1.0: no over-current condition exists
[    2.276915] hub 2-0:1.0: trying to enable port power on non-switchable hub
[    2.277031] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '001'
[    2.277205] xen: registering gsi 17 triggering 0 polarity 1
[    2.277280]   alloc irq_desc for 17 on node 0
[    2.277350]   alloc kstat_irqs on node 0
[    2.277429] xen: --> irq=17
[    2.277499] xen_set_ioapic_routing: irq 17 gsi 17 vector 17 ioapic 0 pin 17 triggering 1 polarity 1
[    2.277615] ohci_hcd 0000:00:13.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
[    2.277702] ohci_hcd 0000:00:13.1: OHCI Host Controller
[    2.277786] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '003'
[    2.278034] ohci_hcd 0000:00:13.1: new USB bus registered, assigned bus number 3
[    2.280993] ohci_hcd 0000:00:13.1: created debug files
[    2.281069] ohci_hcd 0000:00:13.1: supports USB remote wakeup
[    2.281164] ohci_hcd 0000:00:13.1: irq 17, io mem 0xfe02d000
[    2.318029] ehci_hcd 0000:00:13.5: GetStatus port 1 status 001803 POWER sig=j CSC CONNECT
[    2.318142] hub 1-0:1.0: port 1: status 0501 change 0001
[    2.318229] ehci_hcd 0000:00:13.5: GetStatus port 2 status 001803 POWER sig=j CSC CONNECT
[    2.318340] hub 1-0:1.0: port 2: status 0501 change 0001
[    2.335045] ohci_hcd 0000:00:13.1: OHCI controller state
[    2.335118] ohci_hcd 0000:00:13.1: OHCI 1.0, NO legacy support registers
[    2.335191] ohci_hcd 0000:00:13.1: control 0x283 RWC HCFS=operational CBSR=3
[    2.335264] ohci_hcd 0000:00:13.1: cmdstatus 0x00000 SOC=0
[    2.335335] ohci_hcd 0000:00:13.1: intrstatus 0x00000004 SF
[    2.335405] ohci_hcd 0000:00:13.1: intrenable 0x8000001a MIE UE RD WDH
[    2.335483] ohci_hcd 0000:00:13.1: hcca frame #0005
[    2.335553] ohci_hcd 0000:00:13.1: roothub.a 02000202 POTPGT=2 NPS NDP=2(2)
[    2.335624] ohci_hcd 0000:00:13.1: roothub.b 00000000 PPCM=0000 DR=0000
[    2.335695] ohci_hcd 0000:00:13.1: roothub.status 00008000 DRWE
[    2.335766] ohci_hcd 0000:00:13.1: roothub.portstatus [0] 0x00000100 PPS
[    2.335838] ohci_hcd 0000:00:13.1: roothub.portstatus [1] 0x00000100 PPS
[    2.335943] usb usb3: default language 0x0409
[    2.336041] usb usb3: udev 1, busnum 3, minor = 256
[    2.336111] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[    2.336182] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.336292] usb usb3: Product: OHCI Host Controller
[    2.336365] usb usb3: Manufacturer: Linux 2.6.31 ohci_hcd
[    2.336436] usb usb3: SerialNumber: 0000:00:13.1
[    2.336572] usb usb3: uevent
[    2.336736] usb usb3: usb_probe_device
[    2.336811] usb usb3: configuration #1 chosen from 1 choice
[    2.336900] usb usb3: adding 3-0:1.0 (config #1, interface 0)
[    2.337014] usb 3-0:1.0: uevent
[    2.337175] hub 3-0:1.0: usb_probe_interface
[    2.337249] hub 3-0:1.0: usb_probe_interface - got id
[    2.337325] hub 3-0:1.0: USB hub found
[    2.337440] hub 3-0:1.0: 2 ports detected
[    2.337513] hub 3-0:1.0: standalone hub
[    2.337585] hub 3-0:1.0: no power switching (usb 1.0)
[    2.337656] hub 3-0:1.0: global over-current protection
[    2.337727] hub 3-0:1.0: power on to power good time: 4ms
[    2.337818] hub 3-0:1.0: local power source is good
[    2.337892] hub 3-0:1.0: no over-current condition exists
[    2.337964] hub 3-0:1.0: trying to enable port power on non-switchable hub
[    2.338083] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '001'
[    2.338259] xen: registering gsi 18 triggering 0 polarity 1
[    2.338335]   alloc irq_desc for 18 on node 0
[    2.338403]   alloc kstat_irqs on node 0
[    2.338483] xen: --> irq=18
[    2.338552] xen_set_ioapic_routing: irq 18 gsi 18 vector 18 ioapic 0 pin 18 triggering 1 polarity 1
[    2.338668] ohci_hcd 0000:00:13.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
[    2.338755] ohci_hcd 0000:00:13.2: OHCI Host Controller
[    2.338839] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '004'
[    2.339091] ohci_hcd 0000:00:13.2: new USB bus registered, assigned bus number 4
[    2.339251] ohci_hcd 0000:00:13.2: created debug files
[    2.339324] ohci_hcd 0000:00:13.2: supports USB remote wakeup
[    2.339420] ohci_hcd 0000:00:13.2: irq 18, io mem 0xfe02c000
[    2.376052] hub 2-0:1.0: state 7 ports 2 chg 0000 evt 0000
[    2.394045] ohci_hcd 0000:00:13.2: OHCI controller state
[    2.394118] ohci_hcd 0000:00:13.2: OHCI 1.0, NO legacy support registers
[    2.394191] ohci_hcd 0000:00:13.2: control 0x283 RWC HCFS=operational CBSR=3
[    2.394263] ohci_hcd 0000:00:13.2: cmdstatus 0x00000 SOC=0
[    2.394334] ohci_hcd 0000:00:13.2: intrstatus 0x00000004 SF
[    2.394405] ohci_hcd 0000:00:13.2: intrenable 0x8000001a MIE UE RD WDH
[    2.394483] ohci_hcd 0000:00:13.2: hcca frame #0005
[    2.394554] ohci_hcd 0000:00:13.2: roothub.a 02000202 POTPGT=2 NPS NDP=2(2)
[    2.394626] ohci_hcd 0000:00:13.2: roothub.b 00000000 PPCM=0000 DR=0000
[    2.394697] ohci_hcd 0000:00:13.2: roothub.status 00008000 DRWE
[    2.394769] ohci_hcd 0000:00:13.2: roothub.portstatus [0] 0x00000100 PPS
[    2.394841] ohci_hcd 0000:00:13.2: roothub.portstatus [1] 0x00000100 PPS
[    2.394951] usb usb4: default language 0x0409
[    2.395050] usb usb4: udev 1, busnum 4, minor = 384
[    2.395122] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[    2.395193] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.395301] usb usb4: Product: OHCI Host Controller
[    2.395370] usb usb4: Manufacturer: Linux 2.6.31 ohci_hcd
[    2.395439] usb usb4: SerialNumber: 0000:00:13.2
[    2.395579] usb usb4: uevent
[    2.395737] usb usb4: usb_probe_device
[    2.395807] usb usb4: configuration #1 chosen from 1 choice
[    2.395893] usb usb4: adding 4-0:1.0 (config #1, interface 0)
[    2.395992] usb 4-0:1.0: uevent
[    2.396156] hub 4-0:1.0: usb_probe_interface
[    2.396226] hub 4-0:1.0: usb_probe_interface - got id
[    2.396297] hub 4-0:1.0: USB hub found
[    2.396401] hub 4-0:1.0: 2 ports detected
[    2.396472] hub 4-0:1.0: standalone hub
[    2.396542] hub 4-0:1.0: no power switching (usb 1.0)
[    2.396611] hub 4-0:1.0: global over-current protection
[    2.396681] hub 4-0:1.0: power on to power good time: 4ms
[    2.396767] hub 4-0:1.0: local power source is good
[    2.396836] hub 4-0:1.0: no over-current condition exists
[    2.396906] hub 4-0:1.0: trying to enable port power on non-switchable hub
[    2.397026] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '001'
[    2.397197] xen: registering gsi 17 triggering 0 polarity 1
[    2.397269] xen_allocate_pirq: returning irq 17 for gsi 17
[    2.397340] xen: --> irq=17
[    2.397409] xen_set_ioapic_routing: irq 17 gsi 17 vector 17 ioapic 0 pin 17 triggering 1 polarity 1
[    2.397529] ohci_hcd 0000:00:13.3: PCI INT B -> GSI 17 (level, low) -> IRQ 17
[    2.397615] ohci_hcd 0000:00:13.3: OHCI Host Controller
[    2.397700] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '005'
[    2.397938] ohci_hcd 0000:00:13.3: new USB bus registered, assigned bus number 5
[    2.398099] ohci_hcd 0000:00:13.3: created debug files
[    2.398171] ohci_hcd 0000:00:13.3: supports USB remote wakeup
[    2.398248] ohci_hcd 0000:00:13.3: irq 17, io mem 0xfe02b000
[    2.418038] hub 1-0:1.0: state 7 ports 10 chg 0006 evt 0000
[    2.418124] hub 1-0:1.0: port 1, status 0501, change 0000, 480 Mb/s
[    2.453047] ohci_hcd 0000:00:13.3: OHCI controller state
[    2.453122] ohci_hcd 0000:00:13.3: OHCI 1.0, NO legacy support registers
[    2.453194] ohci_hcd 0000:00:13.3: control 0x283 RWC HCFS=operational CBSR=3
[    2.453266] ohci_hcd 0000:00:13.3: cmdstatus 0x00000 SOC=0
[    2.453337] ohci_hcd 0000:00:13.3: intrstatus 0x00000004 SF
[    2.453409] ohci_hcd 0000:00:13.3: intrenable 0x8000001a MIE UE RD WDH
[    2.453487] ohci_hcd 0000:00:13.3: hcca frame #0005
[    2.453559] ohci_hcd 0000:00:13.3: roothub.a 02000202 POTPGT=2 NPS NDP=2(2)
[    2.453630] ohci_hcd 0000:00:13.3: roothub.b 00000000 PPCM=0000 DR=0000
[    2.453701] ohci_hcd 0000:00:13.3: roothub.status 00008000 DRWE
[    2.453774] ohci_hcd 0000:00:13.3: roothub.portstatus [0] 0x00000100 PPS
[    2.453846] ohci_hcd 0000:00:13.3: roothub.portstatus [1] 0x00000100 PPS
[    2.453951] usb usb5: default language 0x0409
[    2.454075] usb usb5: udev 1, busnum 5, minor = 512
[    2.454146] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
[    2.454217] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.454324] usb usb5: Product: OHCI Host Controller
[    2.454393] usb usb5: Manufacturer: Linux 2.6.31 ohci_hcd
[    2.454462] usb usb5: SerialNumber: 0000:00:13.3
[    2.454599] usb usb5: uevent
[    2.454759] usb usb5: usb_probe_device
[    2.454830] usb usb5: configuration #1 chosen from 1 choice
[    2.454915] usb usb5: adding 5-0:1.0 (config #1, interface 0)
[    2.455024] usb 5-0:1.0: uevent
[    2.455181] hub 5-0:1.0: usb_probe_interface
[    2.455252] hub 5-0:1.0: usb_probe_interface - got id
[    2.455323] hub 5-0:1.0: USB hub found
[    2.455427] hub 5-0:1.0: 2 ports detected
[    2.455498] hub 5-0:1.0: standalone hub
[    2.455568] hub 5-0:1.0: no power switching (usb 1.0)
[    2.455637] hub 5-0:1.0: global over-current protection
[    2.455706] hub 5-0:1.0: power on to power good time: 4ms
[    2.455793] hub 5-0:1.0: local power source is good
[    2.455862] hub 5-0:1.0: no over-current condition exists
[    2.455932] hub 5-0:1.0: trying to enable port power on non-switchable hub
[    2.456053] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '001'
[    2.456222] xen: registering gsi 18 triggering 0 polarity 1
[    2.456293] xen_allocate_pirq: returning irq 18 for gsi 18
[    2.456363] xen: --> irq=18
[    2.456432] xen_set_ioapic_routing: irq 18 gsi 18 vector 18 ioapic 0 pin 18 triggering 1 polarity 1
[    2.456550] ohci_hcd 0000:00:13.4: PCI INT C -> GSI 18 (level, low) -> IRQ 18
[    2.456637] ohci_hcd 0000:00:13.4: OHCI Host Controller
[    2.456724] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '006'
[    2.456966] ohci_hcd 0000:00:13.4: new USB bus registered, assigned bus number 6
[    2.457127] ohci_hcd 0000:00:13.4: created debug files
[    2.457199] ohci_hcd 0000:00:13.4: supports USB remote wakeup
[    2.457277] ohci_hcd 0000:00:13.4: irq 18, io mem 0xfe02a000
[    2.469285] ehci_hcd 0000:00:13.5: port 1 high speed
[    2.469360] ehci_hcd 0000:00:13.5: GetStatus port 1 status 001005 POWER sig=se0 PE CONNECT
[    2.512048] ohci_hcd 0000:00:13.4: OHCI controller state
[    2.512121] ohci_hcd 0000:00:13.4: OHCI 1.0, NO legacy support registers
[    2.512194] ohci_hcd 0000:00:13.4: control 0x283 RWC HCFS=operational CBSR=3
[    2.512265] ohci_hcd 0000:00:13.4: cmdstatus 0x00000 SOC=0
[    2.512337] ohci_hcd 0000:00:13.4: intrstatus 0x00000004 SF
[    2.512408] ohci_hcd 0000:00:13.4: intrenable 0x8000001a MIE UE RD WDH
[    2.512486] ohci_hcd 0000:00:13.4: hcca frame #0005
[    2.513003] ohci_hcd 0000:00:13.4: roothub.a 02000202 POTPGT=2 NPS NDP=2(2)
[    2.513003] ohci_hcd 0000:00:13.4: roothub.b 00000000 PPCM=0000 DR=0000
[    2.513003] ohci_hcd 0000:00:13.4: roothub.status 00008000 DRWE
[    2.513003] ohci_hcd 0000:00:13.4: roothub.portstatus [0] 0x00000100 PPS
[    2.513003] ohci_hcd 0000:00:13.4: roothub.portstatus [1] 0x00000100 PPS
[    2.515753] usb usb6: default language 0x0409
[    2.515843] usb usb6: udev 1, busnum 6, minor = 640
[    2.515912] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
[    2.515984] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.516091] usb usb6: Product: OHCI Host Controller
[    2.516160] usb usb6: Manufacturer: Linux 2.6.31 ohci_hcd
[    2.516230] usb usb6: SerialNumber: 0000:00:13.4
[    2.516377] usb usb6: uevent
[    2.516537] usb usb6: usb_probe_device
[    2.516608] usb usb6: configuration #1 chosen from 1 choice
[    2.516693] usb usb6: adding 6-0:1.0 (config #1, interface 0)
[    2.516792] usb 6-0:1.0: uevent
[    2.516948] hub 6-0:1.0: usb_probe_interface
[    2.517019] hub 6-0:1.0: usb_probe_interface - got id
[    2.517089] hub 6-0:1.0: USB hub found
[    2.517201] hub 6-0:1.0: 2 ports detected
[    2.517273] hub 6-0:1.0: standalone hub
[    2.517343] hub 6-0:1.0: no power switching (usb 1.0)
[    2.517412] hub 6-0:1.0: global over-current protection
[    2.517481] hub 6-0:1.0: power on to power good time: 4ms
[    2.517569] hub 6-0:1.0: local power source is good
[    2.517638] hub 6-0:1.0: no over-current condition exists
[    2.517709] hub 6-0:1.0: trying to enable port power on non-switchable hub
[    2.517815] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '001'
[    2.518072] uhci_hcd: USB Universal Host Controller Interface driver
[    2.518548] usbcore: registered new interface driver usblp
[    2.518621] Initializing USB Mass Storage driver...
[    2.518801] usbcore: registered new interface driver usb-storage
[    2.518874] USB Mass Storage support registered.
[    2.519059] usbcore: registered new interface driver libusual
[    2.519508] PNP: No PS/2 controller found. Probing ports directly.
[    2.520155] serio: i8042 KBD port at 0x60,0x64 irq 1
[    2.520232] serio: i8042 AUX port at 0x60,0x64 irq 12
[    2.520291] usb 1-1: new high speed USB device using ehci_hcd and address 2
[    2.520579] mice: PS/2 mouse device common for all mice
[    2.521250] rtc_cmos 00:04: RTC can wake from S4
[    2.521460] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
[    2.521580] rtc0: alarms up to one month, 242 bytes nvram
[    2.521896] it87: Found IT8716F chip at 0x228, revision 1
[    2.521989] it87: in3 is VCC (+5V)
[    2.522062] it87: in7 is VCCH (+5V Stand-By)
[    2.522321] ACPI: I/O resource it87 [0x22d-0x22e] conflicts with ACPI region IP__ [0x22d-0x22e]
[    2.522434] ACPI: Device needs an ACPI driver
[    2.522617] k8temp 0000:00:18.3: Temperature readouts might be wrong - check erratum #141
[    2.522959] md: raid1 personality registered for level 1
[    2.523504] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com
[    2.523857] cpuidle: using governor ladder
[    2.523930] cpuidle: using governor menu
[    2.526630] usbcore: registered new interface driver hiddev
[    2.526804] usbcore: registered new interface driver usbhid
[    2.526876] usbhid: v2.6:USB HID core driver
[    2.527177] Netfilter messages via NETLINK v0.30.
[    2.527282] nf_conntrack version 0.5.0 (4096 buckets, 16384 max)
[    2.527782] ctnetlink v0.93: registering with nfnetlink.
[    2.529421] ip_tables: (C) 2000-2006 Netfilter Core Team
[    2.529530] TCP cubic registered
[    2.529599] Initializing XFRM netlink socket
[    2.530174] NET: Registered protocol family 10
[    2.533243] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    2.533431] IPv6 over IPv4 tunneling driver
[    2.535164] NET: Registered protocol family 17
[    2.535710] RPC: Registered udp transport module.
[    2.535781] RPC: Registered tcp transport module.
[    2.536254] PM: Resume from disk failed.
[    2.536367] registered taskstats version 1
[    2.536602]   Magic number: 9:856:877
[    2.536915] Freeing unused kernel memory: 568k freed
[    2.537304] Write protecting the kernel read-only data: 7344k
[    2.571303] ehci_hcd 0000:00:13.5: port 1 high speed
[    2.571390] ehci_hcd 0000:00:13.5: GetStatus port 1 status 001005 POWER sig=se0 PE CONNECT
[    2.636518] usb 1-1: default language 0x0409
[    2.638893] usb 1-1: udev 2, busnum 1, minor = 1
[    2.638964] usb 1-1: New USB device found, idVendor=1005, idProduct=b113
[    2.639038] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    2.639114] usb 1-1: Product: USB FLASH DRIVE 
[    2.639117] usb 1-1: Manufacturer:         
[    2.639120] usb 1-1: SerialNumber: 1968050104CA
[    2.639231] usb 1-1: uevent
[    2.639532] usb 1-1: usb_probe_device
[    2.639607] usb 1-1: configuration #1 chosen from 1 choice
[    2.639897] usb 1-1: adding 1-1:1.0 (config #1, interface 0)
[    2.640016] usb 1-1:1.0: uevent
[    2.640198] usb-storage 1-1:1.0: usb_probe_interface
[    2.640282] usb-storage 1-1:1.0: usb_probe_interface - got id
[    2.647884] scsi4 : SCSI emulation for USB Mass Storage devices
[    2.656535] usb-storage: device found at 2
[    2.656582] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '002'
[    2.656654] hub 1-0:1.0: port 2, status 0501, change 0000, 480 Mb/s
[    2.656821] usb-storage: waiting for device to settle before scanning
[    2.708046] ehci_hcd 0000:00:13.5: port 2 full speed --> companion
[    2.708137] ehci_hcd 0000:00:13.5: GetStatus port 2 status 003801 POWER OWNER sig=j CONNECT
[    2.708318] hub 1-0:1.0: port 2 not reset yet, waiting 50ms
[    2.760038] ehci_hcd 0000:00:13.5: GetStatus port 2 status 003002 POWER OWNER sig=se0 CSC
[    2.760224] hub 3-0:1.0: state 7 ports 2 chg 0000 evt 0000
[    2.760319] hub 4-0:1.0: state 7 ports 2 chg 0000 evt 0000
[    2.760408] hub 5-0:1.0: state 7 ports 2 chg 0000 evt 0000
[    2.760494] hub 6-0:1.0: state 7 ports 2 chg 0000 evt 0000
[    2.760570] hub 1-0:1.0: state 7 ports 10 chg 0000 evt 0004
[    2.760657] hub 2-0:1.0: state 7 ports 2 chg 0000 evt 0004
[    2.760742] ohci_hcd 0000:00:13.0: GetStatus roothub.portstatus [1] = 0x00010101 CSC PPS CCS
[    2.760873] hub 2-0:1.0: port 2, status 0101, change 0001, 12 Mb/s
[    2.861139] usb usb2: uevent
[    2.861175] usb 2-0:1.0: uevent
[    2.861277] usb usb3: uevent
[    2.861314] usb 3-0:1.0: uevent
[    2.861415] usb usb4: uevent
[    2.861450] usb 4-0:1.0: uevent
[    2.861549] usb usb5: uevent
[    2.861583] usb 5-0:1.0: uevent
[    2.861687] usb usb6: uevent
[    2.861722] usb 6-0:1.0: uevent
[    2.861822] usb usb1: uevent
[    2.861858] usb 1-0:1.0: uevent
[    2.861897] usb 1-1: uevent
[    2.861933] usb 1-1:1.0: uevent
[    2.864048] hub 2-0:1.0: debounce: port 2: total 100ms stable 100ms status 0x101
[    2.885134] ohci_hcd 0000:00:13.0: port[1] reset timeout, stat 00000111
[    2.936150] ohci_hcd 0000:00:13.0: GetStatus roothub.portstatus [1] = 0x00100103 PRSC PPS PES CCS
[    2.987064] usb 2-2: new full speed USB device using ohci_hcd and address 2
[    3.009070] ohci_hcd 0000:00:13.0: port[1] reset timeout, stat 00000113
[    3.060049] ohci_hcd 0000:00:13.0: GetStatus roothub.portstatus [1] = 0x00100103 PRSC PPS PES CCS
[    3.130077] usb 2-2: skipped 1 descriptor after interface
[    3.130082] usb 2-2: skipped 1 descriptor after interface
[    3.134823] usb 2-2: default language 0x0409
[    3.140083] usb 2-2: udev 2, busnum 2, minor = 129
[    3.140164] usb 2-2: New USB device found, idVendor=14dd, idProduct=0002
[    3.140246] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    3.140249] usb 2-2: Product: MultiDevice
[    3.140252] usb 2-2: Manufacturer: Peppercon AG
[    3.140255] usb 2-2: SerialNumber: 768E1175CA002E778E2336515C01B222
[    3.140390] usb 2-2: uevent
[    3.140410] usb 2-2: usb_probe_device
[    3.140414] usb 2-2: configuration #1 chosen from 1 choice
[    3.142084] usb 2-2: adding 2-2:1.0 (config #1, interface 0)
[    3.142125] usb 2-2:1.0: uevent
[    3.142157] usbhid 2-2:1.0: usb_probe_interface
[    3.142161] usbhid 2-2:1.0: usb_probe_interface - got id
[    3.148471] input: Peppercon AG MultiDevice as /devices/pci0000:00/0000:00:13.0/usb2/2-2/2-2:1.0/input/input3
[    3.148709] generic-usb 0003:14DD:0002.0001: input,hidraw0: USB HID v1.01 Keyboard [Peppercon AG MultiDevice] on usb-0000:00:13.0-2/input0
[    3.148875] usb 2-2: adding 2-2:1.1 (config #1, interface 1)
[    3.148983] usb 2-2:1.1: uevent
[    3.149011] usbhid 2-2:1.1: usb_probe_interface
[    3.149116] usbhid 2-2:1.1: usb_probe_interface - got id
[    3.156179] input: Peppercon AG MultiDevice as /devices/pci0000:00/0000:00:13.0/usb2/2-2/2-2:1.1/input/input4
[    3.156442] generic-usb 0003:14DD:0002.0002: input,hidraw1: USB HID v1.01 Mouse [Peppercon AG MultiDevice] on usb-0000:00:13.0-2/input1
[    3.156600] /usr/src/xen/xen-unstable.hg/linux-2.6-pvops.git/drivers/usb/core/inode.c: creating file '002'
[    3.156757] hub 2-0:1.0: state 7 ports 2 chg 0000 evt 0004
[    3.455023] ohci_hcd 0000:00:13.1: auto-stop root hub
[    3.455041] ohci_hcd 0000:00:13.2: auto-stop root hub
[    3.704027] ohci_hcd 0000:00:13.3: auto-stop root hub
[    3.704043] ohci_hcd 0000:00:13.4: auto-stop root hub
[    5.181123] md: md1 stopped.
[    5.192108] md: bind<sdb4>
[    5.192562] md: bind<sda4>
[    5.192995] md: bind<sdc4>
[    5.196365] raid1: raid set md1 active with 3 out of 3 mirrors
[    5.196482] md1: detected capacity change from 0 to 499586957312
[    5.198531]  md1: unknown partition table
[    5.705038] hub 4-0:1.0: hub_suspend
[    5.705127] usb usb4: bus auto-suspend
[    5.705202] ohci_hcd 0000:00:13.2: suspend root hub
[    5.705296] hub 5-0:1.0: hub_suspend
[    5.705373] usb usb5: bus auto-suspend
[    5.705447] ohci_hcd 0000:00:13.3: suspend root hub
[    5.705527] hub 6-0:1.0: hub_suspend
[    5.705604] usb usb6: bus auto-suspend
[    5.705678] ohci_hcd 0000:00:13.4: suspend root hub
[    5.706023] hub 3-0:1.0: hub_suspend
[    5.706101] usb usb3: bus auto-suspend
[    5.706174] ohci_hcd 0000:00:13.1: suspend root hub
[    7.657119] scsi 4:0:0:0: Direct-Access              USB FLASH DRIVE  34CE PQ: 0 ANSI: 0 CCS
[    7.657361] sd 4:0:0:0: Attached scsi generic sg3 type 0
[    7.658235] sd 4:0:0:0: [sdd] 2015232 512-byte logical blocks: (1.03 GB/984 MiB)
[    7.659030] usb-storage: device scan complete
[    7.659462] sd 4:0:0:0: [sdd] Write Protect is off
[    7.659467] sd 4:0:0:0: [sdd] Mode Sense: 23 00 00 00
[    7.659471] sd 4:0:0:0: [sdd] Assuming drive cache: write through
[    7.662974] sd 4:0:0:0: [sdd] Assuming drive cache: write through
[    7.662980]  sdd: sdd1 sdd2 sdd3
[    7.665259] sd 4:0:0:0: [sdd] Assuming drive cache: write through
[    7.665263] sd 4:0:0:0: [sdd] Attached SCSI removable disk
[   12.131313] kjournald starting.  Commit interval 5 seconds
[   12.131332] EXT3-fs: mounted filesystem with writeback data mode.
[   12.852010] eth0: no IPv6 routers present
[   13.381651] udevd version 125 started
[   13.517401] usb usb2: uevent
[   13.517438] usb 2-0:1.0: uevent
[   13.517480] usb 2-2: uevent
[   13.517539] usb 2-2:1.0: uevent
[   13.517788] usb 2-2:1.1: uevent
[   13.518236] usb usb3: uevent
[   13.518273] usb 3-0:1.0: uevent
[   13.518387] usb usb4: uevent
[   13.518424] usb 4-0:1.0: uevent
[   13.518550] usb usb5: uevent
[   13.518598] usb 5-0:1.0: uevent
[   13.518730] usb usb6: uevent
[   13.518767] usb 6-0:1.0: uevent
[   13.518886] usb usb1: uevent
[   13.518937] usb 1-0:1.0

[-- Attachment #3: xen-dmesg --]
[-- Type: text/plain, Size: 8875 bytes --]

 __  __            _____  ____                     _        _     _      
 \ \/ /___ _ __   |___ / | ___|    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ \047_ \    |_ \ |___ \ __| | | | \047_ \/ __| __/ _` | \047_ \| |/ _ \
  /  \  __/ | | |  ___) | ___) |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)____/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                         
(XEN) Xen version 3.5-unstable (root@scharrenberg.com) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) Sat Sep 19 12:38:39 CEST 2009
(XEN) Latest ChangeSet: Fri Sep 18 14:45:40 2009 +0100 20228:3a71e070e3c5
(XEN) Command line: 
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN)  EDID info not retrieved because of reasons unknown
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009f000 (usable)
(XEN)  000000000009f000 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000ddee0000 (usable)
(XEN)  00000000ddee0000 - 00000000ddee3000 (ACPI NVS)
(XEN)  00000000ddee3000 - 00000000ddef0000 (ACPI data)
(XEN)  00000000ddef0000 - 00000000ddf00000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fec00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000220000000 (usable)
(XEN) System RAM: 8158MB (8354300kB)
(XEN) ACPI: RSDP 000F8210, 0024 (r2 ATI   )
(XEN) ACPI: XSDT DDEE3100, 0044 (r1 ATI    ASUSACPI 42302E31 AWRD        0)
(XEN) ACPI: FACP DDEE8500, 00F4 (r3 ATI    ASUSACPI 42302E31 AWRD        0)
(XEN) ACPI: DSDT DDEE3280, 5210 (r1 ATI    ASUSACPI     1000 MSFT  3000000)
(XEN) ACPI: FACS DDEE0000, 0040
(XEN) ACPI: HPET DDEE8740, 0038 (r1 ATI    ASUSACPI 42302E31 AWRD       98)
(XEN) ACPI: MCFG DDEE87C0, 003C (r1 ATI    ASUSACPI 42302E31 AWRD        0)
(XEN) ACPI: APIC DDEE8640, 0084 (r1 ATI    ASUSACPI 42302E31 AWRD        0)
(XEN) NUMA turned off
(XEN) Faking a node at 0000000000000000-0000000220000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000f6560
(XEN) DMI 2.4 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x4008
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[4004,0], pm1x_evt[4000,0]
(XEN) ACPI:                  wakeup_vec[ddee000c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 15:11 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) Processor #1 15:11 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8200 base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
(XEN) PCI: MCFG area at e0000000 reserved in E820
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 2400.019 MHz processor.
(XEN) CPU0: AMD Flush Filter disabled
(XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
(XEN) CPU: L2 Cache: 512K (64 bytes/line)
(XEN) CPU 0(2) -> Core 0
(XEN) AMD SVM: ASIDs disabled. 
(XEN) HVM: SVM enabled
(XEN) CPU0: AMD K8 machine check reporting enabled
(XEN) I/O virtualisation disabled
(XEN) CPU0: AMD Athlon(tm) 64 X2 Dual Core Processor 4600+ stepping 02
(XEN) Booting processor 1/1 eip 8c000
(XEN) Initializing CPU#1
(XEN) CPU1: AMD Flush Filter disabled
(XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
(XEN) CPU: L2 Cache: 512K (64 bytes/line)
(XEN) CPU 1(2) -> Core 1
(XEN) AMD: Disabling C1 Clock Ramping Node #0
(XEN) AMD SVM: ASIDs disabled. 
(XEN) CPU1: AMD K8 machine check reporting enabled
(XEN) CPU1: AMD Athlon(tm) 64 X2 Dual Core Processor 4600+ stepping 02
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) checking TSC synchronization across 2 CPUs: 
(XEN) CPU#0 had -662 usecs TSC skew, fixed it up.
(XEN) CPU#1 had 662 usecs TSC skew, fixed it up.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 2 CPUs
(XEN) do_IRQ: 1.231 No irq handler for vector (irq -1)
(XEN) microcode.c:73:d32767 microcode: CPU1 resumed
(XEN) HPET: 4 timers in total, 0 timers will be used for broadcast
(XEN) ACPI sleep modes: S3
(XEN) MCA: Use hw thresholding to adjust polling frequency
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0xffffffff80200000 memsz=0x26a870
(XEN) elf_parse_binary: phdr: paddr=0xffffffff8046b000 memsz=0x868e8
(XEN) elf_parse_binary: phdr: paddr=0xffffffff804f2000 memsz=0xc08
(XEN) elf_parse_binary: phdr: paddr=0xffffffff804f4000 memsz=0xccc1c
(XEN) elf_parse_binary: memory: 0xffffffff80200000 -> 0xffffffff805c0c1c
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff80200000
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80206000
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: FEATURES = "writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0xffffffff80000000
(XEN)     virt_offset      = 0x0
(XEN)     virt_kstart      = 0xffffffff80200000
(XEN)     virt_kend        = 0xffffffff805c0c1c
(XEN)     virt_entry       = 0xffffffff80200000
(XEN)     p2m_base         = 0xffffffffffffffff
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 0xffffffff805c0c1c
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000214000000->0000000218000000 (2014025 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff805c0c1c
(XEN)  Init. ramdisk: ffffffff805c1000->ffffffff817ec800
(XEN)  Phys-Mach map: ffffffff817ed000->ffffffff8276aa48
(XEN)  Start info:    ffffffff8276b000->ffffffff8276b4b4
(XEN)  Page tables:   ffffffff8276c000->ffffffff82785000
(XEN)  Boot stack:    ffffffff82785000->ffffffff82786000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82c00000
(XEN)  ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 2 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff80200000 -> 0xffffffff8046a870
(XEN) elf_load_binary: phdr 1 at 0xffffffff8046b000 -> 0xffffffff804f18e8
(XEN) elf_load_binary: phdr 2 at 0xffffffff804f2000 -> 0xffffffff804f2c08
(XEN) elf_load_binary: phdr 3 at 0xffffffff804f4000 -> 0xffffffff8052dff8
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type \047CTRL-a\047 three times to switch input to Xen)
(XEN) Freed 140kB init memory.
(XEN) io_apic.c:2206: 
(XEN) ioapic_guest_write: apic=0, pin=2, irq=0
(XEN) ioapic_guest_write: new_entry=00000900
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) PCI add device 00:14.1
(XEN) Set CPU acpi_id(0) cpuid(0) Px State info:
(XEN) 	_PPC: 0
(XEN) Set CPU acpi_id(1) cpuid(1) Px State info:
(XEN) 	_PPC: 0
(XEN) no cpu_id for acpi_id 2
(XEN) no cpu_id for acpi_id 3
(XEN) PCI add device 00:13.0
(XEN) allocated vector for irq:16
(XEN) PCI add device 00:13.1
(XEN) allocated vector for irq:17
(XEN) PCI add device 00:13.2
(XEN) allocated vector for irq:18
(XEN) PCI add device 00:13.3
(XEN) PCI add device 00:13.4
(XEN) PCI add device 00:13.5
(XEN) allocated vector for irq:19
(XEN) PCI add device 00:14.1
(XEN) PCI add device 02:00.0
(XEN) PCI add device 00:12.0
(XEN) allocated vector for irq:22

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvops: AHCI problems with SB600

[Konrad Rzeszutek Wilk]

On Tue, Sep 22, 2009 at 11:00:23AM +0200, Patrick Scharrenberg wrote:
> Konrad Rzeszutek Wilk wrote:
> > Ooooh. Can you give some more details on the hardware flavor? CPU? Memory?
> Its an AMD system on an ASUS M2A-VM mainboard with an Athlon 64 X2 CPU
> and 8GB of memory.
> 
> > What happens if you use dom0_mem=max:512MB or pass in 'irqpoll' to the
> > Linux kernel?
> irqpoll does not change anything but the memory limit does the trick.
> The system then boots fine, I attached the dmesg log.

Patrick,

Lets try a couple of more options, if this isn't too much of a trouble?
Mainly change the dom0_mem=max:512MB to dom0_mem=max:4GB and then dom0_mem=max:6GB.

You ought to have no trouble at 4GB. 6GB will fail if this is a DMA related
issue.

> Since the netconsole now also works, I assume some kind of IRQ problem
> which is strengthened by the fact, that the keyboard didn't work either
> (for scrolling and searching for problems) without the memory limit option.

The log says:
> [    2.209240] ehci_hcd 0000:00:13.5: applying AMD SB600/SB700 USB freeze workaround

That might be the reason your keyboard didn't work. Unless your keyboard is PS/2?

> 
> > OK. I need the kernel boot log to figure out what might be a potential problem.
> > Let's try to get your netconsole working or your serial console? Does this
> > machine even have a serial port? Maybe you can use that to capture the kernel
> > output?
> 
> I attached the bootlog with the 512 memory-limit enabled and the xen dmesg.

Thank you.
> 
> Unfortunately I have to drive 200km to install a serial cable to the
> system. So if there is an IRQ problem would the serial console really
> help or would it suffer from the same irq problem?
>

If it is an IRQ problem it would probably mess up the serial console right as it
is happening. But there are ways to make Xen not do IRQs for serial console.

Here is what I have:
xen.gz  com1=115200,8n1,0xcf00,0  console=com1,vga guest_loglvl=all  dom0_mem=max:512MB

The last argument defines the IRQ. If it is set to '0' it will poll the device instead of
depending on the PCI card to provide the output. The 0xcf00 is the non-standard location
of the COM1 (most systems have that on 0x3f8 or so).

Looking at NewEgg for your motherboard it looks to _not_ have any serial output.
You would need to get a PCI Serial card for this thought.

Re: pvops: AHCI problems with SB600

[Patrick Scharrenberg]

Hi,

> Lets try a couple of more options, if this isn't too much of a trouble?
That's fine with me.

> You ought to have no trouble at 4GB. 6GB will fail if this is a DMA related
> issue.
It's exactly as you said. 4GB works, 6GB fails.

> The log says:
>> [    2.209240] ehci_hcd 0000:00:13.5: applying AMD SB600/SB700 USB freeze workaround
> That might be the reason your keyboard didn't work. Unless your
keyboard is PS/2?

I also tested the same kernel on an SB700 machine and there it works
just fine including SATA and USB.
The keyboard is a remote management card and is indeed connected via USB.

> Looking at NewEgg for your motherboard it looks to _not_ have any serial output.
> You would need to get a PCI Serial card for this thought.

The board has a serial port but no external connector.
I have a serial port bracket which I can install if it helps on.

Is there anything else I can try meanwhile?

Re: pvops: AHCI problems with SB600

[Konrad Rzeszutek Wilk]

On Wed, Sep 23, 2009 at 09:37:55AM +0200, Patrick Scharrenberg wrote:
> Hi,
> 
> > Lets try a couple of more options, if this isn't too much of a trouble?
> That's fine with me.
> 
> > You ought to have no trouble at 4GB. 6GB will fail if this is a DMA related
> > issue.
> It's exactly as you said. 4GB works, 6GB fails.
> 
> > The log says:
> >> [    2.209240] ehci_hcd 0000:00:13.5: applying AMD SB600/SB700 USB freeze workaround
> > That might be the reason your keyboard didn't work. Unless your
> keyboard is PS/2?
> 
> I also tested the same kernel on an SB700 machine and there it works
> just fine including SATA and USB.

Weird. I have a machine with SB700 and if it has more than 4GB it fails.
Your SB700 machine does have more than 4GB, right?

> The keyboard is a remote management card and is indeed connected via USB.

Nice. Just curious - what kind of card is this?
> 
> > Looking at NewEgg for your motherboard it looks to _not_ have any serial output.
> > You would need to get a PCI Serial card for this thought.
> 
> The board has a serial port but no external connector.
> I have a serial port bracket which I can install if it helps on.
> 
> Is there anything else I can try meanwhile?

Let me start working on the machine I have here to troubleshoot.
I would suggest that in the meantime you use the dom0_mem workaround
and see if there are other problems besides the one you found?

Re: pvops: AHCI problems with SB600

[Konrad Rzeszutek Wilk]

On Tue, Sep 22, 2009 at 10:08:25AM -0400, Konrad Rzeszutek Wilk wrote:
> On Tue, Sep 22, 2009 at 11:00:23AM +0200, Patrick Scharrenberg wrote:
> > Konrad Rzeszutek Wilk wrote:
> > > Ooooh. Can you give some more details on the hardware flavor? CPU? Memory?
> > Its an AMD system on an ASUS M2A-VM mainboard with an Athlon 64 X2 CPU
> > and 8GB of memory.
> > 
> > > What happens if you use dom0_mem=max:512MB or pass in 'irqpoll' to the
> > > Linux kernel?
> > irqpoll does not change anything but the memory limit does the trick.
> > The system then boots fine, I attached the dmesg log.
> 
> Patrick,
> 
> Lets try a couple of more options, if this isn't too much of a trouble?
> Mainly change the dom0_mem=max:512MB to dom0_mem=max:4GB and then dom0_mem=max:6GB.
> 
> You ought to have no trouble at 4GB. 6GB will fail if this is a DMA related
> issue.

Patrick,

I've gotten my hands on machine with SB700 and it exhibits similar problems. The 
SB700 AHCI controller stops working if I have more than 4GB in the machine.

Re: pvops: AHCI problems with SB600

[Jeremy Fitzhardinge]

On 09/23/09 05:06, Konrad Rzeszutek Wilk wrote:
> I've gotten my hands on machine with SB700 and it exhibits similar problems. The 
> SB700 AHCI controller stops working if I have more than 4GB in the machine.
>   

You mean just with this kernel?  I presume it works OK normally.

    J

Re: pvops: AHCI problems with SB600

[Konrad Rzeszutek Wilk]

On Wed, Sep 23, 2009 at 12:22:32PM -0700, Jeremy Fitzhardinge wrote:
> On 09/23/09 05:06, Konrad Rzeszutek Wilk wrote:
> > I've gotten my hands on machine with SB700 and it exhibits similar problems. The 
> > SB700 AHCI controller stops working if I have more than 4GB in the machine.
> >   
> 
> You mean just with this kernel?  I presume it works OK normally.

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1514
has the details.

It looks that the calls to ioremap_nocache return an address
that is not synchronized with the physical address.

This is exhibited only when dom0 has more than 4GB, so if you do dom_mem=max:4GB
the machine boots succesfully.

If I boot the machine without Xen, with Linux seeing 8GB, 4GB, 6GB, or whatnot, it
boots succesfully.

Re: pvops: AHCI problems with SB600

[Jeremy Fitzhardinge]

On 09/23/09 12:32, Konrad Rzeszutek Wilk wrote:
> On Wed, Sep 23, 2009 at 12:22:32PM -0700, Jeremy Fitzhardinge wrote:
>   
>> On 09/23/09 05:06, Konrad Rzeszutek Wilk wrote:
>>     
>>> I've gotten my hands on machine with SB700 and it exhibits similar problems. The 
>>> SB700 AHCI controller stops working if I have more than 4GB in the machine.
>>>   
>>>       
>> You mean just with this kernel?  I presume it works OK normally.
>>     
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1514
> has the details.
>
> It looks that the calls to ioremap_nocache return an address
> that is not synchronized with the physical address.
>   

Synchronized in what sense?  ioremap_* is all common code, so I would
expect it to fail or not fail.  I guess the possibilities are that that
the physaddr is getting truncated to 32-bits somewhere, or _PAGE_PCD is
getting masked out (either from __supported_pte_flags, or elsewhere in
the process).

But you mention in the bug that the appears to be problems with the AGP
aperture.  Do you get the "Aperture pointing to e820 RAM. Ignoring"
message when booting native?

We use the real BIOS-provided e820 map and then trim it according to the
provided memory size, so there should always be the same holes in the
E820 map that BIOS provides.

arch/x86/kernel/aperture_64.c has the test:

		if (!no_iommu &&
		    max_pfn > MAX_DMA32_PFN &&
		    !printed_gart_size_msg) {
			printk(KERN_ERR "you are using iommu with agp, but GART size is less than 64M\n");
			printk(KERN_ERR "please increase GART size in your BIOS setup\n");
			printk(KERN_ERR "if BIOS doesn't have that option, contact your HW vendor!\n");
			printed_gart_size_msg = 1;
		}

I guess the "!no_iommu" clause is triggering because we have swiotlb
set, but I wonder if that's specifically testing for the presence of a
GART IOMMU?

How does ioremap_nocache fit into this?  Is the mapping failing in some
way and causing this code to fail?  Or am I misunderstanding?

> This is exhibited only when dom0 has more than 4GB, so if you do dom_mem=max:4GB
> the machine boots succesfully.
>   

The test above also tests "max_pfn > MAX_DMA32_PFN" which limiting
memory would avoid.

    J

Re: pvops: AHCI problems with SB600

[Jeremy Fitzhardinge]

On 09/23/09 13:09, Jeremy Fitzhardinge wrote:
> 		if (!no_iommu &&
> 		    max_pfn > MAX_DMA32_PFN &&
> 		    !printed_gart_size_msg) {
> 			printk(KERN_ERR "you are using iommu with agp, but GART size is less than 64M\n");
> 			printk(KERN_ERR "please increase GART size in your BIOS setup\n");
> 			printk(KERN_ERR "if BIOS doesn't have that option, contact your HW vendor!\n");
> 			printed_gart_size_msg = 1;
> 		}
>   

Oh, that's the wrong error message, but the other one has similar
predicates.  Hm, but it also skips the test if (swiotlb && !valid_agp)...

> I guess the "!no_iommu" clause is triggering because we have swiotlb
> set, but I wonder if that's specifically testing for the presence of a
> GART IOMMU?
>
> How does ioremap_nocache fit into this?  Is the mapping failing in some
> way and causing this code to fail?  Or am I misunderstanding?
>
>   
>> This is exhibited only when dom0 has more than 4GB, so if you do dom_mem=max:4GB
>> the machine boots succesfully.
>>   
>>     
> The test above also tests "max_pfn > MAX_DMA32_PFN" which limiting
> memory would avoid.
>   

    J

Re: pvops: AHCI problems with SB600

[Konrad Rzeszutek Wilk]

On Wed, Sep 23, 2009 at 01:30:44PM -0700, Jeremy Fitzhardinge wrote:
> On 09/23/09 13:09, Jeremy Fitzhardinge wrote:
> > 		if (!no_iommu &&
> > 		    max_pfn > MAX_DMA32_PFN &&
> > 		    !printed_gart_size_msg) {
> > 			printk(KERN_ERR "you are using iommu with agp, but GART size is less than 64M\n");
> > 			printk(KERN_ERR "please increase GART size in your BIOS setup\n");
> > 			printk(KERN_ERR "if BIOS doesn't have that option, contact your HW vendor!\n");
> > 			printed_gart_size_msg = 1;
> > 		}
> >   
> 
> Oh, that's the wrong error message, but the other one has similar
> predicates.  Hm, but it also skips the test if (swiotlb && !valid_agp)...

Right. We don't set the swiotlb. The reason being if you do set it then
the original SWIOTLB kicks in.

The weird part is that the function you copied-n-pasted (gart_iommu_hole_init)
only detectes and allocates a buffer. It does not set the dma_ops at all.
Setting of the dma_ops is done via the gart_iommu_init() call which is done
much later. But with Xen-SWIOTLB already initialized, the gart_iommu_init()
quits right away.

So the kernel sets the dma_ops to the Xen SWIOTLB, and it
allocates an extra 64MB chunk of memory for the GART, which is not
used, and ... somehow all of the ioremap_nocache functions stop working
correctly. Maybe the ioremap_nocache does use some of that memory that
the gart_iommu_hole_init allocated?

With this patch, the GART is forcefully disabled, and the kernel boots fine
(with 6GB, 8GB, etc).

diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index 6b76948..1101a9f 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -122,6 +122,8 @@ void __init pci_iommu_alloc(void)
 	 * The order of these functions is important for
 	 * fall-back/fail-over reasons
 	 */
+	xen_swiotlb_init();
+
 	gart_iommu_hole_init();
 
 	detect_calgary();
@@ -130,8 +132,6 @@ void __init pci_iommu_alloc(void)
 
 	amd_iommu_detect();
 
-	xen_swiotlb_init();
-
 	pci_swiotlb_init();
 }
 
diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
index 5e2c856..00f2260 100644
--- a/arch/x86/xen/pci-swiotlb.c
+++ b/arch/x86/xen/pci-swiotlb.c
@@ -43,6 +43,10 @@
 #include <xen/page.h>
 #include <xen/xen-ops.h>
 
+
+#include <linux/pci.h>
+#include <asm/gart.h>
+
 #define OFFSET(val,align) ((unsigned long)	\
 	                   ( (val) & ( (align) - 1)))
 
@@ -985,5 +989,9 @@ void __init xen_swiotlb_init(void)
 		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default to 64MB */
 		dma_ops = &xen_swiotlb_dma_ops;
 		iommu_detected = 1;
+#ifdef CONFIG_GART_IOMMU
+		gart_iommu_aperture_disabled = 1;
+#endif
+
 	}
 }

Re: pvops: AHCI problems with SB600

[Jeremy Fitzhardinge]

On 09/23/09 14:24, Konrad Rzeszutek Wilk wrote:
> The weird part is that the function you copied-n-pasted (gart_iommu_hole_init)
> only detectes and allocates a buffer. It does not set the dma_ops at all.
> Setting of the dma_ops is done via the gart_iommu_init() call which is done
> much later. But with Xen-SWIOTLB already initialized, the gart_iommu_init()
> quits right away.
>   

Perhaps the fix is to make gart_iommu_hole_init() quit if iommu_detected
too?  Though it is called after xen_swiotlb_init()...

> So the kernel sets the dma_ops to the Xen SWIOTLB, and it
> allocates an extra 64MB chunk of memory for the GART, which is not
> used, and ... somehow all of the ioremap_nocache functions stop working
> correctly. Maybe the ioremap_nocache does use some of that memory that
> the gart_iommu_hole_init allocated?
>   
Can't see how it would affect it.  ioremap allocates a new virtual space
for the mapping and then just plugs in the pfns for the pages you want
to map.  They end up getting _PAGE_IOMAP set in the pte flags, which
causes the xen/mmu.c backend to use the address as-is (ie, as an mfn),
so the mapping will be constructed properly.  Well, that's the theory;
but I'd expect we'd be seeing a lot more havok of ioremap is either
mapping the wrong pages or using the wrong caching.

> With this patch, the GART is forcefully disabled, and the kernel boots fine
> (with 6GB, 8GB, etc).
>   

OK, I'll put it in for now.  Will we have issues with other forms of iommu?

Another thought, could we actually use the gart iommu instead of swiotlb
if it is available?  I think it leads to exactly the same set of issues
as extending normal swiotlb for Xen's use (ie, inserting pfn->mfn
conversion into the correct places, and perhaps allocating the memory
properly).  Worth thinking about; it may shine light on better ways to
fix up swiotlb.

Thanks,
    J

Re: pvops: AHCI problems with SB600

[Konrad Rzeszutek Wilk]

On Wed, Sep 23, 2009 at 02:56:16PM -0700, Jeremy Fitzhardinge wrote:
> On 09/23/09 14:24, Konrad Rzeszutek Wilk wrote:
> > The weird part is that the function you copied-n-pasted (gart_iommu_hole_init)
> > only detectes and allocates a buffer. It does not set the dma_ops at all.
> > Setting of the dma_ops is done via the gart_iommu_init() call which is done
> > much later. But with Xen-SWIOTLB already initialized, the gart_iommu_init()
> > quits right away.
> >   
> 
> Perhaps the fix is to make gart_iommu_hole_init() quit if iommu_detected
> too?  Though it is called after xen_swiotlb_init()...

That is a good idea too. That would avoid that ugly #ifdef.
> 
> > So the kernel sets the dma_ops to the Xen SWIOTLB, and it
> > allocates an extra 64MB chunk of memory for the GART, which is not
> > used, and ... somehow all of the ioremap_nocache functions stop working
> > correctly. Maybe the ioremap_nocache does use some of that memory that
> > the gart_iommu_hole_init allocated?
> >   
> Can't see how it would affect it.  ioremap allocates a new virtual space
> for the mapping and then just plugs in the pfns for the pages you want
> to map.  They end up getting _PAGE_IOMAP set in the pte flags, which
> causes the xen/mmu.c backend to use the address as-is (ie, as an mfn),
> so the mapping will be constructed properly.  Well, that's the theory;
> but I'd expect we'd be seeing a lot more havok of ioremap is either
> mapping the wrong pages or using the wrong caching.

There was a lot of havoc - all of the PCI BARs were useless. Is the MFN
(from the pfn_to_mfn on this address) suppose to have a specific value?

> 
> > With this patch, the GART is forcefully disabled, and the kernel boots fine
> > (with 6GB, 8GB, etc).
> >   
> 
> OK, I'll put it in for now.  Will we have issues with other forms of iommu?

There are three other types: AMD IOMMU (a real IOMMU), Intel's IOMMU,
and the IBM's Calgary IOMMU.

For all of those setting, no_iommu=1 should do the trick. But in reality
I need to double-check that:


diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
index 00f2260..390f698 100644
--- a/arch/x86/xen/pci-swiotlb.c
+++ b/arch/x86/xen/pci-swiotlb.c
@@ -989,6 +989,8 @@ void __init xen_swiotlb_init(void)
 		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default to 64MB */
 		dma_ops = &xen_swiotlb_dma_ops;
 		iommu_detected = 1;
+		no_iommu = 1; /* Forces the other IOMMU (if they are detected) to
+				to quit, rather than initialize. */
 #ifdef CONFIG_GART_IOMMU
 		gart_iommu_aperture_disabled = 1;
 #endif

<sigh>I think I need to rethink this swiotlb-Xen part. This is starting
to look like a hack.

> 
> Another thought, could we actually use the gart iommu instead of swiotlb
> if it is available?  I think it leads to exactly the same set of issues
> as extending normal swiotlb for Xen's use (ie, inserting pfn->mfn
> conversion into the correct places, and perhaps allocating the memory
> properly).  Worth thinking about; it may shine light on better ways to
> fix up swiotlb.

Yes! That was my next step - see if it is possible to use it and if so
extend it for that purpose (and without any ghastly #ifdef).

Re: pvops: AHCI problems with SB600

[Jeremy Fitzhardinge]

On 09/24/09 05:44, Konrad Rzeszutek Wilk wrote:
> There was a lot of havoc - all of the PCI BARs were useless. Is the MFN
> (from the pfn_to_mfn on this address) suppose to have a specific value?
>   

Not sure.  pfn_to_mfn is never supposed to happen on ioremap phys addrs,
because of _PAGE_IOMAP in the pte.  Its probably worth checking that
_PAGE_IOMAP is actually getting set.

> For all of those setting, no_iommu=1 should do the trick. But in reality
> I need to double-check that:
>
>
> diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
> index 00f2260..390f698 100644
> --- a/arch/x86/xen/pci-swiotlb.c
> +++ b/arch/x86/xen/pci-swiotlb.c
> @@ -989,6 +989,8 @@ void __init xen_swiotlb_init(void)
>  		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default to 64MB */
>  		dma_ops = &xen_swiotlb_dma_ops;
>  		iommu_detected = 1;
> +		no_iommu = 1; /* Forces the other IOMMU (if they are detected) to
> +				to quit, rather than initialize. */
>  #ifdef CONFIG_GART_IOMMU
>  		gart_iommu_aperture_disabled = 1;
>  #endif
>
> <sigh>I think I need to rethink this swiotlb-Xen part. This is starting
> to look like a hack.
>   

It isn't great.  We need a way to either layer or arbitrate between
these different address translation mechanisms.

>> Another thought, could we actually use the gart iommu instead of swiotlb
>> if it is available?  I think it leads to exactly the same set of issues
>> as extending normal swiotlb for Xen's use (ie, inserting pfn->mfn
>> conversion into the correct places, and perhaps allocating the memory
>> properly).  Worth thinking about; it may shine light on better ways to
>> fix up swiotlb.
>>     
> Yes! That was my next step - see if it is possible to use it and if so
> extend it for that purpose (and without any ghastly #ifdef).
>   

Good.

    J

Re: pvops: AHCI problems with SB600

[Konrad Rzeszutek Wilk]

On Thu, Sep 24, 2009 at 11:23:16AM -0700, Jeremy Fitzhardinge wrote:
> On 09/24/09 05:44, Konrad Rzeszutek Wilk wrote:
> > There was a lot of havoc - all of the PCI BARs were useless. Is the MFN
> > (from the pfn_to_mfn on this address) suppose to have a specific value?
> >   
> 
> Not sure.  pfn_to_mfn is never supposed to happen on ioremap phys addrs,
> because of _PAGE_IOMAP in the pte.  Its probably worth checking that
> _PAGE_IOMAP is actually getting set.

<sigh> I found the problem. I did not power off the machine after doing
a non-Xen boot where the GART was used. I am not entirely sure what
the GART was doing, but pretty much all writew/readw were not doing/going
where they were suppose to.

> 
> > For all of those setting, no_iommu=1 should do the trick. But in reality
> > I need to double-check that:
> >
> >
> > diff --git a/arch/x86/xen/pci-swiotlb.c b/arch/x86/xen/pci-swiotlb.c
> > index 00f2260..390f698 100644
> > --- a/arch/x86/xen/pci-swiotlb.c
> > +++ b/arch/x86/xen/pci-swiotlb.c
> > @@ -989,6 +989,8 @@ void __init xen_swiotlb_init(void)
> >  		xen_swiotlb_init_with_default_size(64 * (1<<20));	/* default to 64MB */
> >  		dma_ops = &xen_swiotlb_dma_ops;
> >  		iommu_detected = 1;
> > +		no_iommu = 1; /* Forces the other IOMMU (if they are detected) to
> > +				to quit, rather than initialize. */
> >  #ifdef CONFIG_GART_IOMMU
> >  		gart_iommu_aperture_disabled = 1;
> >  #endif
> >
> > <sigh>I think I need to rethink this swiotlb-Xen part. This is starting
> > to look like a hack.
> >   
> 
> It isn't great.  We need a way to either layer or arbitrate between
> these different address translation mechanisms.

I am sending another patch that I think is more nicer, and it takes care of the other IOMMUs.
> 
> >> Another thought, could we actually use the gart iommu instead of swiotlb
> >> if it is available?  I think it leads to exactly the same set of issues
> >> as extending normal swiotlb for Xen's use (ie, inserting pfn->mfn
> >> conversion into the correct places, and perhaps allocating the memory
> >> properly).  Worth thinking about; it may shine light on better ways to
> >> fix up swiotlb.
> >>     
> > Yes! That was my next step - see if it is possible to use it and if so
> > extend it for that purpose (and without any ghastly #ifdef).
> >   
> 
> Good.

Still thinking about this ..

Re: pvops: AHCI problems with SB600

[Patrick Scharrenberg]

Hi Konrad,

> With this patch, the GART is forcefully disabled, and the kernel boots fine
> (with 6GB, 8GB, etc).

I tried your patch and my SB600 system also boots fine. Great. Thanks!
Now I'm going to do some tests on this kernel the next days and will report.

Patrick

Re: Announcing xen/master: pvops git trees rearranged (AMD tests?)

[Konrad Rzeszutek Wilk]

. snip ..

> This is definitely a work-in-progress kernel.  I'd appreciate all bug
> *and* success reports so I can get some idea of how many people are
> using this thing, and how often there are problems.  Patches gratefully
> accepted.

If you have an AMD machine with AGP and 4GB of memory a test of this
tree would be much appreciated. The test matrix is lacking those sorely
and if you see any weird issues like USB not working or ping not working
anymore, please do report it. Or as Jeremy said, also success reports.

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge wrote:
> 
> This is definitely a work-in-progress kernel.  I'd appreciate all bug
> *and* success reports so I can get some idea of how many people are
> using this thing, and how often there are problems. 
>

I just compiled latest xen/master tree and it seems to work for me. 
I was able to run CentOS 5.3, Fedora 11 and Fedora 12 PV domUs without
problems.

My dom0 is currently Fedora 12 (rawhide) running Xen 3.4.1-4 rpms from
the distro.

dom0 kernel is 32bit PAE, CONFIG_HIGHPTE=n.

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Jeremy Fitzhardinge]

On 09/21/09 12:25, Pasi Kärkkäinen wrote:
> I just compiled latest xen/master tree and it seems to work for me. 
> I was able to run CentOS 5.3, Fedora 11 and Fedora 12 PV domUs without
> problems.
>
> My dom0 is currently Fedora 12 (rawhide) running Xen 3.4.1-4 rpms from
> the distro.
>
> dom0 kernel is 32bit PAE, CONFIG_HIGHPTE=n.
>   

Thanks for the feedback.  What hardware do you have?

    J

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Mon, Sep 21, 2009 at 12:30:10PM -0700, Jeremy Fitzhardinge wrote:
> On 09/21/09 12:25, Pasi Kärkkäinen wrote:
> > I just compiled latest xen/master tree and it seems to work for me. 
> > I was able to run CentOS 5.3, Fedora 11 and Fedora 12 PV domUs without
> > problems.
> >
> > My dom0 is currently Fedora 12 (rawhide) running Xen 3.4.1-4 rpms from
> > the distro.
> >
> > dom0 kernel is 32bit PAE, CONFIG_HIGHPTE=n.
> >   
> 
> Thanks for the feedback.  What hardware do you have?
>

This is the same old evil P4 with hyperthreading.. :)

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Jeremy Fitzhardinge]

On 09/21/09 12:50, Pasi Kärkkäinen wrote:
> This is the same old evil P4 with hyperthreading.. :)
>   

What kind of disk and net devices again?

    J

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Mon, Sep 21, 2009 at 01:21:05PM -0700, Jeremy Fitzhardinge wrote:
> On 09/21/09 12:50, Pasi Kärkkäinen wrote:
> > This is the same old evil P4 with hyperthreading.. :)
> >   
> 
> What kind of disk and net devices again?
> 

Intel ICH6 controller (ata_piix), IDE disk, and tg3 gigabit NICs: 
Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Jeremy Fitzhardinge]

On 09/21/09 13:26, Pasi Kärkkäinen wrote:
> Intel ICH6 controller (ata_piix), IDE disk, and tg3 gigabit NICs: 
> Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
>   

So pre-AHCI?

    J

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Mon, Sep 21, 2009 at 01:29:55PM -0700, Jeremy Fitzhardinge wrote:
> On 09/21/09 13:26, Pasi Kärkkäinen wrote:
> > Intel ICH6 controller (ata_piix), IDE disk, and tg3 gigabit NICs: 
> > Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 11)
> >   
> 
> So pre-AHCI?
> 

It has also AHCI controller onboard, but I don't have currently any SATA
disks plugged into the controller.. I think ahci module is loaded.

I might be able to test some AHCI/SATA setup later this week..

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Christian Tramnitz]

Hi Jeremy

are there plans to get any of the (non-bugfix) changes to upstream?
I think the 2.6.32 merge window will close very soon right?


Best regards,
    Christian

Re: Re: Announcing xen/master: pvops git trees rearranged

[Jeremy Fitzhardinge]

On 09/23/09 06:16, Christian Tramnitz wrote:
> are there plans to get any of the (non-bugfix) changes to upstream?
> I think the 2.6.32 merge window will close very soon right?

No plans to put anything into .32.  We need to have a solid story about
how to handle IOAPIC setup before pushing the rest, I think.  I've just
restarted work on that, but I need to work out how to reconcile it with
the recent MSI work.

    J

Re: Re: Announcing xen/master: pvops git trees rearranged

[Qing He]

On Thu, 2009-09-24 at 04:13 +0800, Jeremy Fitzhardinge wrote:
> On 09/23/09 06:16, Christian Tramnitz wrote:
> > are there plans to get any of the (non-bugfix) changes to upstream?
> > I think the 2.6.32 merge window will close very soon right?
> 
> No plans to put anything into .32.  We need to have a solid story about
> how to handle IOAPIC setup before pushing the rest, I think.  I've just
> restarted work on that, but I need to work out how to reconcile it with
> the recent MSI work.
> 
>     J

What's your current thinking on IOAPIC setup, still similar to that
new-routing branch? I'd like to know about it so the MSI can adapt.

And for a trap and emulation based MSI, the current logic may have
to be changed. Currently I'm still trying to find an elegant solution,
so I think you can do the IOAPIC work without much burden of MSI.

Thanks,
Qing

Re: Re: Announcing xen/master: pvops git trees rearranged

[Jeremy Fitzhardinge]

On 09/23/09 20:17, Qing He wrote:
> What's your current thinking on IOAPIC setup, still similar to that
> new-routing branch? I'd like to know about it so the MSI can adapt.
>   

Yes, that's my goal.

> And for a trap and emulation based MSI, the current logic may have
> to be changed. Currently I'm still trying to find an elegant solution,
> so I think you can do the IOAPIC work without much burden of MSI.
>   

OK.  I still need to resolve some conflicts between new-routing and your
MSI-related changes though.

    J

RE: Re: Announcing xen/master: pvops git trees rearranged

[Zhang, Xiantao]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

 Jeremy, 
   After reading your branch new_interrupt_routing,  I think the main changes about two hypercalls and their purposes maybe unnecessary.  I also implemented the similar logic to remove ioapic changes from pv_ops dom0 and just re-used and extended existing interfaces for that.  As to the new-introduced hypercall PHYSDEVOP_route_gsi,  the existing hypercall PHYSDEVOP_map_pirq can cover its functionality through some extension. And for the hypercall PHYSDEVOP_acpi_irq_model, seems it is redundant and unncessary, because irq_model can be parsed through the related acpi tables, so hypervisor and dom0 can reach the agreement automatically after parsing the tables.   

The attached two patches are based on latest Xen and pv_ops_dom0, and they should works for you with latest Xen and pv_ops dom0. Since they are only for furuther discussion, so one dirty hack about probe_gsi also exists in the current code.  
Xiantao

-----Original Message-----
From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] 
Sent: Thursday, September 24, 2009 4:14 AM
To: Christian Tramnitz
Cc: xen-devel@lists.xensource.com; He, Qing; Zhang, Xiantao
Subject: Re: [Xen-devel] Re: Announcing xen/master: pvops git trees rearranged

On 09/23/09 06:16, Christian Tramnitz wrote:
> are there plans to get any of the (non-bugfix) changes to upstream?
> I think the 2.6.32 merge window will close very soon right?

No plans to put anything into .32.  We need to have a solid story about
how to handle IOAPIC setup before pushing the rest, I think.  I've just
restarted work on that, but I need to work out how to reconcile it with
the recent MSI work.

    J

[-- Attachment #2: pv_ops_dom.patch --]
[-- Type: application/octet-stream, Size: 10088 bytes --]

diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
index 70f5ea9..b45dedf 100644
--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -188,9 +188,5 @@ static inline void probe_nr_irqs_gsi(void)	{ }
 #endif
 
 void xen_io_apic_init(void);
-unsigned int xen_io_apic_read(unsigned apic, unsigned reg);
-void xen_io_apic_write(unsigned int apic,
-		       unsigned int reg, unsigned int value);
-
 
 #endif /* _ASM_X86_IO_APIC_H */
diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
index d47c54f..783678c 100644
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -954,6 +954,9 @@ int __init acpi_probe_gsi(void)
 	int idx;
 	int gsi;
 	int max_gsi = 0;
+    
+	if (xen_initial_domain()) /* Dirty hack!! */
+        return 256;
 
 	if (acpi_disabled)
 		return 0;
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 77151ce..0c87499 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -386,9 +386,6 @@ static inline unsigned int io_apic_read(unsigned int apic, unsigned int reg)
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain())
-		return xen_io_apic_read(apic, reg);
-
 	io_apic = io_apic_base(apic);
 	writel(reg, &io_apic->index);
 	return readl(&io_apic->data);
@@ -398,11 +395,6 @@ static inline void io_apic_write(unsigned int apic, unsigned int reg, unsigned i
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_write(apic, reg, value);
-		return;
-	}
-
 	io_apic = io_apic_base(apic);
 	writel(reg, &io_apic->index);
 	writel(value, &io_apic->data);
@@ -418,11 +410,6 @@ static inline void io_apic_modify(unsigned int apic, unsigned int reg, unsigned
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_write(apic, reg, value);
-		return;
-	}
-
 	io_apic = io_apic_base(apic);
 
 	if (sis_apic_bug)
@@ -3162,9 +3149,6 @@ static int __init ioapic_init_sysfs(void)
 	struct sys_device * dev;
 	int i, size, error;
 
-	if (xen_initial_domain())
-		return 0;
-
 	error = sysdev_class_register(&ioapic_sysdev_class);
 	if (error)
 		return error;
@@ -4183,11 +4167,6 @@ void __init ioapic_init_mappings(void)
 	struct resource *ioapic_res;
 	int i;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_init();
-		return;
-	}
-
 	ioapic_res = ioapic_setup_resources();
 	for (i = 0; i < nr_ioapics; i++) {
 		if (smp_found_config) {
diff --git a/arch/x86/xen/apic.c b/arch/x86/xen/apic.c
index ee0db39..805ce70 100644
--- a/arch/x86/xen/apic.c
+++ b/arch/x86/xen/apic.c
@@ -17,31 +17,6 @@ void __init xen_io_apic_init(void)
 	enable_IO_APIC();
 }
 
-unsigned int xen_io_apic_read(unsigned apic, unsigned reg)
-{
-	struct physdev_apic apic_op;
-	int ret;
-
-	apic_op.apic_physbase = mp_ioapics[apic].apicaddr;
-	apic_op.reg = reg;
-	ret = HYPERVISOR_physdev_op(PHYSDEVOP_apic_read, &apic_op);
-	if (ret)
-		BUG();
-	return apic_op.value;
-}
-
-
-void xen_io_apic_write(unsigned int apic, unsigned int reg, unsigned int value)
-{
-	struct physdev_apic apic_op;
-
-	apic_op.apic_physbase = mp_ioapics[apic].apicaddr;
-	apic_op.reg = reg;
-	apic_op.value = value;
-	if (HYPERVISOR_physdev_op(PHYSDEVOP_apic_write, &apic_op))
-		BUG();
-}
-
 void xen_init_apic(void)
 {
 	if (!xen_initial_domain())
diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index a654a49..0ca0c2c 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1938,6 +1938,8 @@ __init pgd_t *xen_setup_kernel_pagetable(pgd_t *pgd,
 }
 #endif	/* CONFIG_X86_64 */
 
+static unsigned char dummy_ioapic_mapping[PAGE_SIZE] __page_aligned_bss;
+
 static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)
 {
 	pte_t pte;
@@ -1973,7 +1975,7 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)
 		 * We just don't map the IO APIC - all access is via
 		 * hypercalls.  Keep the address in the pte for reference.
 		 */
-		pte = pfn_pte(phys, PAGE_NONE);
+		pte = __pte(__pa(dummy_ioapic_mapping) | __PAGE_KERNEL);
 		break;
 #endif
 	case FIX_PARAVIRT_BOOTMAP:
diff --git a/arch/x86/xen/pci.c b/arch/x86/xen/pci.c
index 44d91ad..9ec0d08 100644
--- a/arch/x86/xen/pci.c
+++ b/arch/x86/xen/pci.c
@@ -15,52 +15,38 @@
 
 #include "xen-ops.h"
 
-static void xen_set_io_apic_routing(int irq, int trigger, int polarity)
-{
-	int ioapic, ioapic_pin;
-	int vector, gsi;
-	struct IO_APIC_route_entry entry;
-
-	gsi = xen_gsi_from_irq(irq);
-	vector = xen_vector_from_irq(irq);
-
-	ioapic = mp_find_ioapic(gsi);
-	if (ioapic == -1) {
-		printk(KERN_WARNING "xen_set_ioapic_routing: irq %d gsi %d ioapic %d\n",
-			irq, gsi, ioapic);
-		return;
-	}
-
-	ioapic_pin = mp_find_ioapic_pin(ioapic, gsi);
-
-	printk(KERN_INFO "xen_set_ioapic_routing: irq %d gsi %d vector %d ioapic %d pin %d triggering %d polarity %d\n",
-		irq, gsi, vector, ioapic, ioapic_pin, trigger, polarity);
-
-	setup_ioapic_entry(ioapic, -1, &entry, ~0, trigger, polarity, vector,
-			   ioapic_pin);
-	ioapic_write_entry(ioapic, ioapic_pin, entry);
-}
-
 int xen_register_gsi(u32 gsi, int triggering, int polarity)
 {
-	int irq;
+	int rc, irq;
+	struct physdev_map_pirq map_irq;
+	domid_t domid = DOMID_SELF;
 
 	if (!xen_domain())
 		return -1;
 
-	printk(KERN_DEBUG "xen: registering gsi %u triggering %d polarity %d\n",
-	       gsi, triggering, polarity);
+	BUG_ON(gsi >= (1 << 16));
 
 	irq = xen_allocate_pirq(gsi, (triggering == ACPI_EDGE_SENSITIVE)
-				     ? "ioapic-edge" : "ioapic-level");
+					 ? "ioapic-edge" : "ioapic-level");
 
 	printk(KERN_DEBUG "xen: --> irq=%d\n", irq);
 
-	if (irq >= 0)
-		xen_set_io_apic_routing(irq,
-					triggering == ACPI_EDGE_SENSITIVE ? 0 : 1,
-					polarity == ACPI_ACTIVE_HIGH ? 0 : 1);
-
+	triggering = (triggering == ACPI_EDGE_SENSITIVE ? 0 : 1);
+	polarity =  (polarity == ACPI_ACTIVE_HIGH ? 0 : 1);
+	printk( "xen: registering gsi %u triggering %d polarity %d\n",
+		   gsi, triggering, polarity);
+
+	memset(&map_irq, 0, sizeof(map_irq));
+	map_irq.domid = domid;
+	map_irq.type = MAP_PIRQ_TYPE_GSI;
+	map_irq.index = gsi | (triggering << 16) | (polarity << 24);
+	map_irq.pirq = irq;
+
+	rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);
+	if (rc) {
+		printk(KERN_WARNING "xen map irq failed %d\n", rc);
+		irq = -1;
+	}
 	return irq;
 }
 
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 68c287c..52e4294 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -74,7 +74,7 @@ enum xen_irq_type {
  * event channel - irq->event channel mapping
  * cpu - cpu this event channel is bound to
  * index - type-specific information:
- *    PIRQ - vector, with MSB being "needs EIO"
+ *    PIRQ - with MSB being "needs EIO"
  *    VIRQ - virq number
  *    IPI - IPI vector
  *    EVTCHN -
@@ -90,7 +90,6 @@ struct irq_info
 		enum ipi_vector ipi;
 		struct {
 			unsigned short nr;
-			unsigned char vector;
 			unsigned char flags;
 		} pirq;
 	} u;
@@ -144,11 +143,10 @@ static struct irq_info mk_virq_info(unsigned short evtchn, unsigned short virq)
 			.cpu = 0, .u.virq = virq };
 }
 
-static struct irq_info mk_pirq_info(unsigned short evtchn,
-				    unsigned short pirq, unsigned short vector)
+static struct irq_info mk_pirq_info(unsigned short evtchn, unsigned short pirq)
 {
 	return (struct irq_info) { .type = IRQT_PIRQ, .evtchn = evtchn,
-			.cpu = 0, .u.pirq = { .nr = pirq, .vector = vector } };
+			.cpu = 0, .u.pirq = { .nr = pirq } };
 }
 
 /*
@@ -170,16 +168,6 @@ unsigned irq_from_evtchn(unsigned int evtchn)
 }
 EXPORT_SYMBOL_GPL(irq_from_evtchn);
 
-static enum ipi_vector ipi_from_irq(unsigned irq)
-{
-	struct irq_info *info = info_for_irq(irq);
-
-	BUG_ON(info == NULL);
-	BUG_ON(info->type != IRQT_IPI);
-
-	return info->u.ipi;
-}
-
 static unsigned virq_from_irq(unsigned irq)
 {
 	struct irq_info *info = info_for_irq(irq);
@@ -200,16 +188,6 @@ static unsigned gsi_from_irq(unsigned irq)
 	return info->u.pirq.nr;
 }
 
-static unsigned vector_from_irq(unsigned irq)
-{
-	struct irq_info *info = info_for_irq(irq);
-
-	BUG_ON(info == NULL);
-	BUG_ON(info->type != IRQT_PIRQ);
-
-	return info->u.pirq.vector;
-}
-
 static enum xen_irq_type type_from_irq(unsigned irq)
 {
 	return info_for_irq(irq)->type;
@@ -539,14 +517,13 @@ static int find_irq_by_gsi(unsigned gsi)
 }
 
 /*
- * Allocate a physical irq, along with a vector.  We don't assign an
+ * Allocate a physical irq.  We don't assign an
  * event channel until the irq actually started up.  Return an
  * existing irq if we've already got one for the gsi.
  */
 int xen_allocate_pirq(unsigned gsi, char *name)
 {
 	int irq;
-	struct physdev_irq irq_op;
 
 	spin_lock(&irq_mapping_update_lock);
 
@@ -567,14 +544,7 @@ int xen_allocate_pirq(unsigned gsi, char *name)
 	set_irq_chip_and_handler_name(irq, &xen_pirq_chip,
 				      handle_level_irq, name);
 
-	irq_op.irq = gsi;
-	if (HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
-		dynamic_irq_cleanup(irq);
-		irq = -ENOSPC;
- 		goto out;
- 	}
-
-	irq_info[irq] = mk_pirq_info(0, gsi, irq_op.vector);
+	irq_info[irq] = mk_pirq_info(0, gsi);
 out:
 	spin_unlock(&irq_mapping_update_lock);
 	return irq;
@@ -657,7 +627,7 @@ int xen_create_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc, int type)
 		goto out;
 	}
 
-	irq_info[irq] = mk_pirq_info(0, map_irq.pirq, map_irq.index);
+	irq_info[irq] = mk_pirq_info(0, map_irq.pirq);
 	set_irq_chip_and_handler_name(irq, &xen_pirq_chip,
 			handle_level_irq,
 			(type == PCI_CAP_ID_MSIX) ? "msi-x":"msi");
@@ -668,11 +638,6 @@ out:
 }
 #endif
 
-int xen_vector_from_irq(unsigned irq)
-{
-	return vector_from_irq(irq);
-}
-
 int xen_gsi_from_irq(unsigned irq)
 {
 	return gsi_from_irq(irq);
@@ -752,6 +717,15 @@ static int bind_interdomain_evtchn_to_irq(unsigned int remote_domain,
         return err ? : bind_evtchn_to_irq(bind_interdomain.local_port);
 }
 
+static enum ipi_vector ipi_from_irq(unsigned irq)
+{
+	struct irq_info *info = info_for_irq(irq);
+
+	BUG_ON(info == NULL);
+	BUG_ON(info->type != IRQT_IPI);
+
+	return info->u.ipi;
+}
 
 int bind_virq_to_irq(unsigned int virq, unsigned int cpu)
 {

[-- Attachment #3: xen.patch --]
[-- Type: application/octet-stream, Size: 4108 bytes --]

diff -r 8f81bdd57afe xen/arch/x86/irq.c
--- a/xen/arch/x86/irq.c	Thu Sep 03 09:51:37 2009 +0100
+++ b/xen/arch/x86/irq.c	Thu Sep 24 15:36:19 2009 +0800
@@ -55,6 +55,11 @@ DEFINE_PER_CPU(vector_irq_t, vector_irq)
 
 DEFINE_PER_CPU(struct cpu_user_regs *, __irq_regs);
 
+int check_irq_status(int irq)
+{
+    return irq_status[irq] != IRQ_UNUSED ? 1 : 0;
+}
+
 /* Must be called when irq disabled */
 void lock_vector_lock(void)
 {
@@ -588,6 +593,9 @@ int setup_irq(unsigned int irq, struct i
     desc->depth   = 0;
     desc->status &= ~IRQ_DISABLED;
     desc->handler->startup(irq);
+
+    if ( !check_irq_status(irq) )
+        irq_status[irq] = IRQ_USED;
 
     spin_unlock_irqrestore(&desc->lock,flags);
 
@@ -1277,6 +1285,8 @@ int map_domain_pirq(
 
     ASSERT(spin_is_locked(&pcidevs_lock));
     ASSERT(spin_is_locked(&d->event_lock));
+    
+    desc = irq_to_desc(irq);
 
     if ( !IS_PRIV(current->domain) )
         return -EPERM;
@@ -1288,6 +1298,13 @@ int map_domain_pirq(
         return -EINVAL;
     }
 
+    if ( desc->action )
+    {
+        dprintk(XENLOG_G_WARNING, "Attempt to map in-use IRQ by Xen,"
+                        " irq:%d!\n", irq);
+        return 0;
+    }
+
     old_irq = domain_pirq_to_irq(d, pirq);
     old_pirq = domain_irq_to_pirq(d, irq);
 
@@ -1307,7 +1324,6 @@ int map_domain_pirq(
         return ret;
     }
 
-    desc = irq_to_desc(irq);
 
     if ( type == MAP_PIRQ_TYPE_MSI )
     {
diff -r 8f81bdd57afe xen/arch/x86/physdev.c
--- a/xen/arch/x86/physdev.c	Thu Sep 03 09:51:37 2009 +0100
+++ b/xen/arch/x86/physdev.c	Thu Sep 24 15:45:17 2009 +0800
@@ -30,7 +30,7 @@ static int physdev_map_pirq(struct physd
 static int physdev_map_pirq(struct physdev_map_pirq *map)
 {
     struct domain *d;
-    int pirq, irq, ret = 0;
+    int pirq = 0, irq, ret = 0;
     struct msi_info _msi;
     void *map_data = NULL;
 
@@ -55,23 +55,28 @@ static int physdev_map_pirq(struct physd
     switch ( map->type )
     {
         case MAP_PIRQ_TYPE_GSI:
-            if ( map->index < 0 || map->index >= nr_irqs_gsi )
+        {
+            int gsi, triggering, polarity;
+            
+            gsi = map->index & 0xffff;
+            triggering = !!(map->index & (1 << 16));
+            polarity = !!(map->index & (1 << 24));
+            irq = pirq = map->pirq;
+            
+            if ( gsi < 0 || gsi >= nr_irqs_gsi )
             {
-                dprintk(XENLOG_G_ERR, "dom%d: map invalid irq %d\n",
-                        d->domain_id, map->index);
+                dprintk(XENLOG_G_ERR, "dom%d: map invalid gsi %d\n",
+                        d->domain_id, gsi);
                 ret = -EINVAL;
                 goto free_domain;
             }
-            irq = domain_pirq_to_irq(current->domain, map->index);
-            if ( !irq )
-            {
-                dprintk(XENLOG_G_ERR, "dom%d: map pirq with incorrect irq!\n",
-                        d->domain_id);
-                ret = -EINVAL;
-                goto free_domain;
-            }
-            break;
-
+            if ( !check_irq_status(irq) ) {
+                mp_register_gsi(gsi, triggering, polarity);
+                printk("Register gsi:%d for dom:%d, irq:%d\n", gsi,
+                      d->domain_id, irq);
+            }
+            break;
+        }
         case MAP_PIRQ_TYPE_MSI:
             irq = map->index;
             if ( irq == -1 )
@@ -103,7 +108,6 @@ static int physdev_map_pirq(struct physd
     spin_lock(&pcidevs_lock);
     /* Verify or get pirq. */
     spin_lock(&d->event_lock);
-    pirq = domain_irq_to_pirq(d, irq);
     if ( map->pirq < 0 )
     {
         if ( pirq )
diff -r 8f81bdd57afe xen/include/asm-x86/irq.h
--- a/xen/include/asm-x86/irq.h	Thu Sep 03 09:51:37 2009 +0100
+++ b/xen/include/asm-x86/irq.h	Sat Sep 05 16:09:07 2009 +0800
@@ -123,6 +123,8 @@ int __assign_irq_vector(int irq, struct 
 
 int bind_irq_vector(int irq, int vector, cpumask_t domain);
 
+int check_irq_status(int irq);
+
 #define domain_pirq_to_irq(d, pirq) ((d)->arch.pirq_irq[pirq])
 #define domain_irq_to_pirq(d, irq) ((d)->arch.irq_pirq[irq])
 

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Re: Announcing xen/master: pvops git trees rearranged

[Zhang, Xiantao]

Inline the pv_ops dom0 patch. 

diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
index 70f5ea9..b45dedf 100644
--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -188,9 +188,5 @@ static inline void probe_nr_irqs_gsi(void)	{ }
 #endif
 
 void xen_io_apic_init(void);
-unsigned int xen_io_apic_read(unsigned apic, unsigned reg);
-void xen_io_apic_write(unsigned int apic,
-		       unsigned int reg, unsigned int value);
-
 
 #endif /* _ASM_X86_IO_APIC_H */
diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
index d47c54f..783678c 100644
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -954,6 +954,9 @@ int __init acpi_probe_gsi(void)
 	int idx;
 	int gsi;
 	int max_gsi = 0;
+    
+	if (xen_initial_domain()) /* Dirty hack!! */
+        return 256;
 
 	if (acpi_disabled)
 		return 0;
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 77151ce..0c87499 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -386,9 +386,6 @@ static inline unsigned int io_apic_read(unsigned int apic, unsigned int reg)
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain())
-		return xen_io_apic_read(apic, reg);
-
 	io_apic = io_apic_base(apic);
 	writel(reg, &io_apic->index);
 	return readl(&io_apic->data);
@@ -398,11 +395,6 @@ static inline void io_apic_write(unsigned int apic, unsigned int reg, unsigned i
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_write(apic, reg, value);
-		return;
-	}
-
 	io_apic = io_apic_base(apic);
 	writel(reg, &io_apic->index);
 	writel(value, &io_apic->data);
@@ -418,11 +410,6 @@ static inline void io_apic_modify(unsigned int apic, unsigned int reg, unsigned
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_write(apic, reg, value);
-		return;
-	}
-
 	io_apic = io_apic_base(apic);
 
 	if (sis_apic_bug)
@@ -3162,9 +3149,6 @@ static int __init ioapic_init_sysfs(void)
 	struct sys_device * dev;
 	int i, size, error;
 
-	if (xen_initial_domain())
-		return 0;
-
 	error = sysdev_class_register(&ioapic_sysdev_class);
 	if (error)
 		return error;
@@ -4183,11 +4167,6 @@ void __init ioapic_init_mappings(void)
 	struct resource *ioapic_res;
 	int i;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_init();
-		return;
-	}
-
 	ioapic_res = ioapic_setup_resources();
 	for (i = 0; i < nr_ioapics; i++) {
 		if (smp_found_config) {
diff --git a/arch/x86/xen/apic.c b/arch/x86/xen/apic.c
index ee0db39..805ce70 100644
--- a/arch/x86/xen/apic.c
+++ b/arch/x86/xen/apic.c
@@ -17,31 +17,6 @@ void __init xen_io_apic_init(void)
 	enable_IO_APIC();
 }
 
-unsigned int xen_io_apic_read(unsigned apic, unsigned reg)
-{
-	struct physdev_apic apic_op;
-	int ret;
-
-	apic_op.apic_physbase = mp_ioapics[apic].apicaddr;
-	apic_op.reg = reg;
-	ret = HYPERVISOR_physdev_op(PHYSDEVOP_apic_read, &apic_op);
-	if (ret)
-		BUG();
-	return apic_op.value;
-}
-
-
-void xen_io_apic_write(unsigned int apic, unsigned int reg, unsigned int value)
-{
-	struct physdev_apic apic_op;
-
-	apic_op.apic_physbase = mp_ioapics[apic].apicaddr;
-	apic_op.reg = reg;
-	apic_op.value = value;
-	if (HYPERVISOR_physdev_op(PHYSDEVOP_apic_write, &apic_op))
-		BUG();
-}
-
 void xen_init_apic(void)
 {
 	if (!xen_initial_domain())
diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index a654a49..0ca0c2c 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1938,6 +1938,8 @@ __init pgd_t *xen_setup_kernel_pagetable(pgd_t *pgd,
 }
 #endif	/* CONFIG_X86_64 */
 
+static unsigned char dummy_ioapic_mapping[PAGE_SIZE] __page_aligned_bss;
+
 static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)
 {
 	pte_t pte;
@@ -1973,7 +1975,7 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)
 		 * We just don't map the IO APIC - all access is via
 		 * hypercalls.  Keep the address in the pte for reference.
 		 */
-		pte = pfn_pte(phys, PAGE_NONE);
+		pte = __pte(__pa(dummy_ioapic_mapping) | __PAGE_KERNEL);
 		break;
 #endif
 	case FIX_PARAVIRT_BOOTMAP:
diff --git a/arch/x86/xen/pci.c b/arch/x86/xen/pci.c
index 44d91ad..9ec0d08 100644
--- a/arch/x86/xen/pci.c
+++ b/arch/x86/xen/pci.c
@@ -15,52 +15,38 @@
 
 #include "xen-ops.h"
 
-static void xen_set_io_apic_routing(int irq, int trigger, int polarity)
-{
-	int ioapic, ioapic_pin;
-	int vector, gsi;
-	struct IO_APIC_route_entry entry;
-
-	gsi = xen_gsi_from_irq(irq);
-	vector = xen_vector_from_irq(irq);
-
-	ioapic = mp_find_ioapic(gsi);
-	if (ioapic == -1) {
-		printk(KERN_WARNING "xen_set_ioapic_routing: irq %d gsi %d ioapic %d\n",
-			irq, gsi, ioapic);
-		return;
-	}
-
-	ioapic_pin = mp_find_ioapic_pin(ioapic, gsi);
-
-	printk(KERN_INFO "xen_set_ioapic_routing: irq %d gsi %d vector %d ioapic %d pin %d triggering %d polarity %d\n",
-		irq, gsi, vector, ioapic, ioapic_pin, trigger, polarity);
-
-	setup_ioapic_entry(ioapic, -1, &entry, ~0, trigger, polarity, vector,
-			   ioapic_pin);
-	ioapic_write_entry(ioapic, ioapic_pin, entry);
-}
-
 int xen_register_gsi(u32 gsi, int triggering, int polarity)
 {
-	int irq;
+	int rc, irq;
+	struct physdev_map_pirq map_irq;
+	domid_t domid = DOMID_SELF;
 
 	if (!xen_domain())
 		return -1;
 
-	printk(KERN_DEBUG "xen: registering gsi %u triggering %d polarity %d\n",
-	       gsi, triggering, polarity);
+	BUG_ON(gsi >= (1 << 16));
 
 	irq = xen_allocate_pirq(gsi, (triggering == ACPI_EDGE_SENSITIVE)
-				     ? "ioapic-edge" : "ioapic-level");
+					 ? "ioapic-edge" : "ioapic-level");
 
 	printk(KERN_DEBUG "xen: --> irq=%d\n", irq);
 
-	if (irq >= 0)
-		xen_set_io_apic_routing(irq,
-					triggering == ACPI_EDGE_SENSITIVE ? 0 : 1,
-					polarity == ACPI_ACTIVE_HIGH ? 0 : 1);
-
+	triggering = (triggering == ACPI_EDGE_SENSITIVE ? 0 : 1);
+	polarity =  (polarity == ACPI_ACTIVE_HIGH ? 0 : 1);
+	printk( "xen: registering gsi %u triggering %d polarity %d\n",
+		   gsi, triggering, polarity);
+
+	memset(&map_irq, 0, sizeof(map_irq));
+	map_irq.domid = domid;
+	map_irq.type = MAP_PIRQ_TYPE_GSI;
+	map_irq.index = gsi | (triggering << 16) | (polarity << 24);
+	map_irq.pirq = irq;
+
+	rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);
+	if (rc) {
+		printk(KERN_WARNING "xen map irq failed %d\n", rc);
+		irq = -1;
+	}
 	return irq;
 }
 
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 68c287c..52e4294 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -74,7 +74,7 @@ enum xen_irq_type {
  * event channel - irq->event channel mapping
  * cpu - cpu this event channel is bound to
  * index - type-specific information:
- *    PIRQ - vector, with MSB being "needs EIO"
+ *    PIRQ - with MSB being "needs EIO"
  *    VIRQ - virq number
  *    IPI - IPI vector
  *    EVTCHN -
@@ -90,7 +90,6 @@ struct irq_info
 		enum ipi_vector ipi;
 		struct {
 			unsigned short nr;
-			unsigned char vector;
 			unsigned char flags;
 		} pirq;
 	} u;
@@ -144,11 +143,10 @@ static struct irq_info mk_virq_info(unsigned short evtchn, unsigned short virq)
 			.cpu = 0, .u.virq = virq };
 }
 
-static struct irq_info mk_pirq_info(unsigned short evtchn,
-				    unsigned short pirq, unsigned short vector)
+static struct irq_info mk_pirq_info(unsigned short evtchn, unsigned short pirq)
 {
 	return (struct irq_info) { .type = IRQT_PIRQ, .evtchn = evtchn,
-			.cpu = 0, .u.pirq = { .nr = pirq, .vector = vector } };
+			.cpu = 0, .u.pirq = { .nr = pirq } };
 }
 
 /*
@@ -170,16 +168,6 @@ unsigned irq_from_evtchn(unsigned int evtchn)
 }
 EXPORT_SYMBOL_GPL(irq_from_evtchn);
 
-static enum ipi_vector ipi_from_irq(unsigned irq)
-{
-	struct irq_info *info = info_for_irq(irq);
-
-	BUG_ON(info == NULL);
-	BUG_ON(info->type != IRQT_IPI);
-
-	return info->u.ipi;
-}
-
 static unsigned virq_from_irq(unsigned irq)
 {
 	struct irq_info *info = info_for_irq(irq);
@@ -200,16 +188,6 @@ static unsigned gsi_from_irq(unsigned irq)
 	return info->u.pirq.nr;
 }
 
-static unsigned vector_from_irq(unsigned irq)
-{
-	struct irq_info *info = info_for_irq(irq);
-
-	BUG_ON(info == NULL);
-	BUG_ON(info->type != IRQT_PIRQ);
-
-	return info->u.pirq.vector;
-}
-
 static enum xen_irq_type type_from_irq(unsigned irq)
 {
 	return info_for_irq(irq)->type;
@@ -539,14 +517,13 @@ static int find_irq_by_gsi(unsigned gsi)
 }
 
 /*
- * Allocate a physical irq, along with a vector.  We don't assign an
+ * Allocate a physical irq.  We don't assign an
  * event channel until the irq actually started up.  Return an
  * existing irq if we've already got one for the gsi.
  */
 int xen_allocate_pirq(unsigned gsi, char *name)
 {
 	int irq;
-	struct physdev_irq irq_op;
 
 	spin_lock(&irq_mapping_update_lock);
 
@@ -567,14 +544,7 @@ int xen_allocate_pirq(unsigned gsi, char *name)
 	set_irq_chip_and_handler_name(irq, &xen_pirq_chip,
 				      handle_level_irq, name);
 
-	irq_op.irq = gsi;
-	if (HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
-		dynamic_irq_cleanup(irq);
-		irq = -ENOSPC;
- 		goto out;
- 	}
-
-	irq_info[irq] = mk_pirq_info(0, gsi, irq_op.vector);
+	irq_info[irq] = mk_pirq_info(0, gsi);
 out:
 	spin_unlock(&irq_mapping_update_lock);
 	return irq;
@@ -657,7 +627,7 @@ int xen_create_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc, int type)
 		goto out;
 	}
 
-	irq_info[irq] = mk_pirq_info(0, map_irq.pirq, map_irq.index);
+	irq_info[irq] = mk_pirq_info(0, map_irq.pirq);
 	set_irq_chip_and_handler_name(irq, &xen_pirq_chip,
 			handle_level_irq,
 			(type == PCI_CAP_ID_MSIX) ? "msi-x":"msi");
@@ -668,11 +638,6 @@ out:
 }
 #endif
 
-int xen_vector_from_irq(unsigned irq)
-{
-	return vector_from_irq(irq);
-}
-
 int xen_gsi_from_irq(unsigned irq)
 {
 	return gsi_from_irq(irq);
@@ -752,6 +717,15 @@ static int bind_interdomain_evtchn_to_irq(unsigned int remote_domain,
         return err ? : bind_evtchn_to_irq(bind_interdomain.local_port);
 }
 
+static enum ipi_vector ipi_from_irq(unsigned irq)
+{
+	struct irq_info *info = info_for_irq(irq);
+
+	BUG_ON(info == NULL);
+	BUG_ON(info->type != IRQT_IPI);
+
+	return info->u.ipi;
+}
 
 int bind_virq_to_irq(unsigned int virq, unsigned int cpu)
 {
 

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Zhang, Xiantao
Sent: Thursday, September 24, 2009 4:15 PM
To: Jeremy Fitzhardinge; Christian Tramnitz
Cc: xen-devel@lists.xensource.com; He, Qing
Subject: RE: [Xen-devel] Re: Announcing xen/master: pvops git trees rearranged

 Jeremy, 
   After reading your branch new_interrupt_routing,  I think the main changes about two hypercalls and their purposes maybe unnecessary.  I also implemented the similar logic to remove ioapic changes from pv_ops dom0 and just re-used and extended existing interfaces for that.  As to the new-introduced hypercall PHYSDEVOP_route_gsi,  the existing hypercall PHYSDEVOP_map_pirq can cover its functionality through some extension. And for the hypercall PHYSDEVOP_acpi_irq_model, seems it is redundant and unncessary, because irq_model can be parsed through the related acpi tables, so hypervisor and dom0 can reach the agreement automatically after parsing the tables.   

The attached two patches are based on latest Xen and pv_ops_dom0, and they should works for you with latest Xen and pv_ops dom0. Since they are only for furuther discussion, so one dirty hack about probe_gsi also exists in the current code.  
Xiantao

-----Original Message-----
From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] 
Sent: Thursday, September 24, 2009 4:14 AM
To: Christian Tramnitz
Cc: xen-devel@lists.xensource.com; He, Qing; Zhang, Xiantao
Subject: Re: [Xen-devel] Re: Announcing xen/master: pvops git trees rearranged

On 09/23/09 06:16, Christian Tramnitz wrote:
> are there plans to get any of the (non-bugfix) changes to upstream?
> I think the 2.6.32 merge window will close very soon right?

No plans to put anything into .32.  We need to have a solid story about
how to handle IOAPIC setup before pushing the rest, I think.  I've just
restarted work on that, but I need to work out how to reconcile it with
the recent MSI work.

    J

APIC rework

[Jeremy Fitzhardinge]

On 10/19/09 04:05, Zhang, Xiantao wrote:
> Attached the patches, and also created the patch for Xen-3.4-testing tree.   Maybe Keir needs to check-in this patch before Xen-3.4.2 to keep new dom0 be able to run on Xen-3.4-2.  Add Keir to the list. :-)
>   

Hi,

I was just looking at this again and I realize there's been some
non-trival changes which cause conflicts with this patch.  Would you
have a chance to revise the patch for the current xen/master? 
Alternatively, would it be possible to base the patch on the
xen/dom0/apic topic branch?  It's possible this is awkward because of
conflicts from the PCI front/back branches, in which case it may need
some extra patches for those branches.

For now I've applied your current patch to the branch
xen/dom0/apic-xiantao, rooted on the last xen/master branch where it
applies cleanly.

Thanks,
    J

Re: APIC rework

[Jeremy Fitzhardinge]

On 11/11/09 16:47, Jeremy Fitzhardinge wrote:
> For now I've applied your current patch to the branch
> xen/dom0/apic-xiantao, rooted on the last xen/master branch where it
> applies cleanly.
>   

And xen/master-xiantao is my attempt to merge it into master.  It turned
out less complex than I'd thought, but I haven't tested it yet.

    J

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/11/09 17:00, Jeremy Fitzhardinge wrote:
> On 11/11/09 16:47, Jeremy Fitzhardinge wrote:
>   
>> For now I've applied your current patch to the branch
>> xen/dom0/apic-xiantao, rooted on the last xen/master branch where it
>> applies cleanly.
>>   
>>     
> And xen/master-xiantao is my attempt to merge it into master.  It turned
> out less complex than I'd thought, but I haven't tested it yet.
>   

And it works, at least for simple stuff (haven't tried passthrough or
MSI yet).

Keir, how do you feel about this change for Xen?

x86: Change the interface physdev_map_pirq to support new dom0.

It also keeps compatibility with old dom0. 

Signed-off-by: Xiantao Zhang <xiantao.zhang@intel.com>

diff -r 8f81bdd57afe xen/arch/x86/irq.c
--- a/xen/arch/x86/irq.c	Thu Sep 03 09:51:37 2009 +0100
+++ b/xen/arch/x86/irq.c	Thu Sep 24 15:36:19 2009 +0800
@@ -55,6 +55,11 @@ DEFINE_PER_CPU(vector_irq_t, vector_irq)
 
 DEFINE_PER_CPU(struct cpu_user_regs *, __irq_regs);
 
+int check_irq_status(int irq)
+{
+    return irq_status[irq] != IRQ_UNUSED ? 1 : 0;
+}
+
 /* Must be called when irq disabled */
 void lock_vector_lock(void)
 {
@@ -588,6 +593,9 @@ int setup_irq(unsigned int irq, struct i
     desc->depth   = 0;
     desc->status &= ~IRQ_DISABLED;
     desc->handler->startup(irq);
+
+    if ( !check_irq_status(irq) )
+        irq_status[irq] = IRQ_USED;
 
     spin_unlock_irqrestore(&desc->lock,flags);
 
@@ -1277,6 +1285,8 @@ int map_domain_pirq(
 
     ASSERT(spin_is_locked(&pcidevs_lock));
     ASSERT(spin_is_locked(&d->event_lock));
+    
+    desc = irq_to_desc(irq);
 
     if ( !IS_PRIV(current->domain) )
         return -EPERM;
@@ -1288,6 +1298,13 @@ int map_domain_pirq(
         return -EINVAL;
     }
 
+    if ( desc->action )
+    {
+        dprintk(XENLOG_G_WARNING, "Attempt to map in-use IRQ by Xen,"
+                        " irq:%d!\n", irq);
+        return 0;
+    }
+
     old_irq = domain_pirq_to_irq(d, pirq);
     old_pirq = domain_irq_to_pirq(d, irq);
 
@@ -1307,7 +1324,6 @@ int map_domain_pirq(
         return ret;
     }
 
-    desc = irq_to_desc(irq);
 
     if ( type == MAP_PIRQ_TYPE_MSI )
     {
diff -r 8f81bdd57afe xen/arch/x86/physdev.c
--- a/xen/arch/x86/physdev.c	Thu Sep 03 09:51:37 2009 +0100
+++ b/xen/arch/x86/physdev.c	Thu Sep 24 15:45:17 2009 +0800
@@ -30,7 +30,7 @@ static int physdev_map_pirq(struct physd
 static int physdev_map_pirq(struct physdev_map_pirq *map)
 {
     struct domain *d;
-    int pirq, irq, ret = 0;
+    int pirq = 0, irq, ret = 0;
     struct msi_info _msi;
     void *map_data = NULL;
 
@@ -55,23 +55,28 @@ static int physdev_map_pirq(struct physd
     switch ( map->type )
     {
         case MAP_PIRQ_TYPE_GSI:
-            if ( map->index < 0 || map->index >= nr_irqs_gsi )
+        {
+            int gsi, triggering, polarity;
+            
+            gsi = map->index & 0xffff;
+            triggering = !!(map->index & (1 << 16));
+            polarity = !!(map->index & (1 << 24));
+            irq = pirq = map->pirq;
+            
+            if ( gsi < 0 || gsi >= nr_irqs_gsi )
             {
-                dprintk(XENLOG_G_ERR, "dom%d: map invalid irq %d\n",
-                        d->domain_id, map->index);
+                dprintk(XENLOG_G_ERR, "dom%d: map invalid gsi %d\n",
+                        d->domain_id, gsi);
                 ret = -EINVAL;
                 goto free_domain;
             }
-            irq = domain_pirq_to_irq(current->domain, map->index);
-            if ( !irq )
-            {
-                dprintk(XENLOG_G_ERR, "dom%d: map pirq with incorrect irq!\n",
-                        d->domain_id);
-                ret = -EINVAL;
-                goto free_domain;
-            }
-            break;
-
+            if ( !check_irq_status(irq) ) {
+                mp_register_gsi(gsi, triggering, polarity);
+                printk("Register gsi:%d for dom:%d, irq:%d\n", gsi,
+                      d->domain_id, irq);
+            }
+            break;
+        }
         case MAP_PIRQ_TYPE_MSI:
             irq = map->index;
             if ( irq == -1 )
@@ -103,7 +108,6 @@ static int physdev_map_pirq(struct physd
     spin_lock(&pcidevs_lock);
     /* Verify or get pirq. */
     spin_lock(&d->event_lock);
-    pirq = domain_irq_to_pirq(d, irq);
     if ( map->pirq < 0 )
     {
         if ( pirq )
diff -r 8f81bdd57afe xen/include/asm-x86/irq.h
--- a/xen/include/asm-x86/irq.h	Thu Sep 03 09:51:37 2009 +0100
+++ b/xen/include/asm-x86/irq.h	Sat Sep 05 16:09:07 2009 +0800
@@ -123,6 +123,8 @@ int __assign_irq_vector(int irq, struct 
 
 int bind_irq_vector(int irq, int vector, cpumask_t domain);
 
+int check_irq_status(int irq);
+
 #define domain_pirq_to_irq(d, pirq) ((d)->arch.pirq_irq[pirq])
 #define domain_irq_to_pirq(d, irq) ((d)->arch.irq_pirq[irq])
 

	J

RE: Re: APIC rework

[Zhang, Xiantao]

Jeremy Fitzhardinge wrote:
> On 11/11/09 17:00, Jeremy Fitzhardinge wrote:
>> On 11/11/09 16:47, Jeremy Fitzhardinge wrote:
>> 
>>> For now I've applied your current patch to the branch
>>> xen/dom0/apic-xiantao, rooted on the last xen/master branch where
>>> it applies cleanly. 
>>> 
>>> 
>> And xen/master-xiantao is my attempt to merge it into master.  It
>> turned out less complex than I'd thought, but I haven't tested it
>> yet. 
>> 
> 
> And it works, at least for simple stuff (haven't tried passthrough or
> MSI yet).

Hi, Jeremy
   I had tested VT-d and MSI when made the patch.  But anyway, we had better do a full functional testing since the code base varies a lot.  :)
Xiantao

Re: Re: APIC rework

[Keir Fraser]

On 12/11/2009 23:51, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:

> nd it works, at least for simple stuff (haven't tried passthrough or
> MSI yet).
> 
> Keir, how do you feel about this change for Xen?

Sure, I'll check it in.

 -- Keir

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/12/09 23:24, Keir Fraser wrote:
> On 12/11/2009 23:51, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:
>
>   
>> nd it works, at least for simple stuff (haven't tried passthrough or
>> MSI yet).
>>
>> Keir, how do you feel about this change for Xen?
>>     
> Sure, I'll check it in.
>   

How would you feel about backporting this into a stable release?  If we
start using this interrupt setup mechanism it will be a red-letter day
for dom0/xen compatibility.

    J

Re: Re: APIC rework

[Keir Fraser]

On 13/11/2009 23:57, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:

>> Sure, I'll check it in.
>>   
> 
> How would you feel about backporting this into a stable release?  If we
> start using this interrupt setup mechanism it will be a red-letter day
> for dom0/xen compatibility.

There probably won't be another stable release until 4.0 now. There will be
a 3.4.3 about the same time though.

 -- Keir

RE: Re: APIC rework

[Zhang, Xiantao]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

Hi, Keir/Jeremy
   After you picking the two patches into upstream, we found it may break old dom0 when assigned a level-triggered devices to a HVM domain.  The casue is that, old dom0 can't provide trigger mode and polarity when they do map_domain_pirq.  In attached patches, they introduce a bit to indicate whether old dom0 or not.  

xen-introduce-a-bit-to-identify-dom0-type.patch:  for hypervisor. 
0001-x86-Introduce-a-bit-MAP_COMPAT-mode-for-MAP_PIRQ_TY.patch: for pv ops dom0. 

Xiantao


Keir Fraser wrote:
> On 13/11/2009 23:57, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:
> 
>>> Sure, I'll check it in.
>>> 
>> 
>> How would you feel about backporting this into a stable release?  If
>> we start using this interrupt setup mechanism it will be a
>> red-letter day for dom0/xen compatibility.
> 
> There probably won't be another stable release until 4.0 now. There
> will be a 3.4.3 about the same time though.
> 
>  -- Keir


[-- Attachment #2: 0001-x86-Introduce-a-bit-MAP_COMPAT-mode-for-MAP_PIRQ_TY.patch --]
[-- Type: application/octet-stream, Size: 1742 bytes --]

From 151f4aa8a300051f97000445009ab36485175c5e Mon Sep 17 00:00:00 2001
From: Xiantao Zhang <xiantao.zhang@intel.com>
Date: Mon, 16 Nov 2009 17:10:10 -0500
Subject: [PATCH] x86: Introduce a bit MAP_COMPAT mode for MAP_PIRQ_TYPE_GSI.

Always Set this bit for pv_ops dom0, and this bit can make
old dom0 run latest Xen hypervisor, since hypervisor
can leverage this bit to identify old dom0.

Signed-off-by: Xiantao Zhang <xiantao.zhang@intel.com>
---
 arch/x86/xen/pci.c              |    3 ++-
 include/xen/interface/physdev.h |    5 +++++
 2 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/arch/x86/xen/pci.c b/arch/x86/xen/pci.c
index 4cb2fc6..0ecb775 100644
--- a/arch/x86/xen/pci.c
+++ b/arch/x86/xen/pci.c
@@ -48,7 +48,8 @@ int xen_register_gsi(u32 gsi, int triggering, int polarity)
 	memset(&map_irq, 0, sizeof(map_irq));
 	map_irq.domid = domid;
 	map_irq.type = MAP_PIRQ_TYPE_GSI;
-	map_irq.index = gsi | (triggering << 16) | (polarity << 24);
+	map_irq.index = gsi | (triggering << MAP_TRIGGER_BIT) |
+        (polarity << MAP_POLARITY_BIT) | (1 << MAP_COMPAT_BIT);
 	map_irq.pirq = irq;
 
 	rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);
diff --git a/include/xen/interface/physdev.h b/include/xen/interface/physdev.h
index ac5de37..9cc99d9 100644
--- a/include/xen/interface/physdev.h
+++ b/include/xen/interface/physdev.h
@@ -110,6 +110,11 @@ struct physdev_irq {
 #define MAP_PIRQ_TYPE_GSI		0x1
 #define MAP_PIRQ_TYPE_UNKNOWN		0x2
 
+#define MAP_RESERVE_BIT                 31 
+#define MAP_COMPAT_BIT                  30
+#define MAP_TRIGGER_BIT                 16
+#define MAP_POLARITY_BIT                24
+
 #define PHYSDEVOP_map_pirq		13
 struct physdev_map_pirq {
     domid_t domid;
-- 
1.6.0.rc1


[-- Attachment #3: xen-introduce-a-bit-to-identify-dom0-type.patch --]
[-- Type: application/octet-stream, Size: 4417 bytes --]

# HG changeset patch
# User Xiantao Zhang <xiantao.zhang@intel.com>
# Date 1258362094 -28800
# Node ID 3a0d7cb68eae4a4cf4f26474fdb4d9ef35fe4ddf
# Parent  bec27eb6f72cfb3fcecd8eb3060869fa301a62df
x86: Introduce a bit in map->index to indicate dom0's type.

To compat with 2.6.18 dom0, introduce a bit in map->index
to indicate whether dom0 is old dom0(2.6.18) or pv_ops dom0.
pv_ops dom0 should always set this bit for GSI map.

Signed-off-by: Xiantao Zhang <xiantao.zhang@intel.com>

diff -r bec27eb6f72c -r 3a0d7cb68eae xen/arch/x86/irq.c
--- a/xen/arch/x86/irq.c	Sat Nov 14 10:32:59 2009 +0000
+++ b/xen/arch/x86/irq.c	Mon Nov 16 17:01:34 2009 +0800
@@ -1427,13 +1427,6 @@ int map_domain_pirq(
         return -EINVAL;
     }
 
-    if ( desc->action )
-    {
-        dprintk(XENLOG_G_WARNING, "Attempt to map in-use IRQ by Xen,"
-                        " irq:%d!\n", irq);
-        return 0;
-    }
-
     old_irq = domain_pirq_to_irq(d, pirq);
     old_pirq = domain_irq_to_pirq(d, irq);
 
@@ -1452,7 +1445,6 @@ int map_domain_pirq(
                 d->domain_id, pirq);
         return ret;
     }
-
 
     if ( type == MAP_PIRQ_TYPE_MSI )
     {
diff -r bec27eb6f72c -r 3a0d7cb68eae xen/arch/x86/physdev.c
--- a/xen/arch/x86/physdev.c	Sat Nov 14 10:32:59 2009 +0000
+++ b/xen/arch/x86/physdev.c	Mon Nov 16 17:01:34 2009 +0800
@@ -56,25 +56,49 @@ static int physdev_map_pirq(struct physd
     {
         case MAP_PIRQ_TYPE_GSI:
         {
-            int gsi, triggering, polarity;
-            
-            gsi = map->index & 0xffff;
-            triggering = !!(map->index & (1 << 16));
-            polarity = !!(map->index & (1 << 24));
-            irq = pirq = map->pirq;
-            
-            if ( gsi < 0 || gsi >= nr_irqs_gsi )
+            int compat_mode;
+
+            if ( map->index < 0 ) 
             {
-                dprintk(XENLOG_G_ERR, "dom%d: map invalid gsi %d\n",
-                        d->domain_id, gsi);
+                dprintk(XENLOG_G_ERR, "dom%d: map invalid irq %d\n",
+                        d->domain_id, map->index);
                 ret = -EINVAL;
                 goto free_domain;
             }
-            if ( !check_irq_status(irq) ) {
-                mp_register_gsi(gsi, triggering, polarity);
-                printk("Register gsi:%d for dom:%d, irq:%d\n", gsi,
-                      d->domain_id, irq);
+            
+            compat_mode = !(map->index & (1 << MAP_COMPAT_BIT));
+            if (!compat_mode) {
+                int gsi, triggering, polarity;
+                gsi = map->index & 0xffff;
+                triggering = !!(map->index & (1 << MAP_TRIGGER_BIT));
+                polarity = !!(map->index & (1 << MAP_POLARITY_BIT));
+
+                irq = map->pirq;
+
+                if ( gsi < 0 || gsi >= nr_irqs_gsi )
+                {
+                    dprintk(XENLOG_G_ERR, "dom%d: map invalid gsi %d\n",
+                            d->domain_id, gsi);
+                    ret = -EINVAL;
+                    goto free_domain;
+                }
+                if ( !check_irq_status(irq) ) {
+                    mp_register_gsi(gsi, triggering, polarity);
+                    printk("Register gsi:%d for dom:%d, irq:%d\n", gsi,
+                            d->domain_id, irq);
+                }
+            } else {
+
+                irq = domain_pirq_to_irq(current->domain, map->index);
+                if ( !irq )
+                {
+                    dprintk(XENLOG_G_ERR, "dom%d: map pirq with incorrect irq!\n",
+                            d->domain_id);
+                    ret = -EINVAL;
+                    goto free_domain;
+                }
             }
+            pirq = domain_irq_to_pirq(d, irq);
             break;
         }
         case MAP_PIRQ_TYPE_MSI:
diff -r bec27eb6f72c -r 3a0d7cb68eae xen/include/public/physdev.h
--- a/xen/include/public/physdev.h	Sat Nov 14 10:32:59 2009 +0000
+++ b/xen/include/public/physdev.h	Mon Nov 16 17:01:34 2009 +0800
@@ -143,6 +143,12 @@ DEFINE_XEN_GUEST_HANDLE(physdev_irq_t);
 #define MAP_PIRQ_TYPE_GSI               0x1
 #define MAP_PIRQ_TYPE_UNKNOWN           0x2
 
+
+#define MAP_RESERVE_BIT                 31 
+#define MAP_COMPAT_BIT                  30
+#define MAP_TRIGGER_BIT                 16
+#define MAP_POLARITY_BIT                24
+
 #define PHYSDEVOP_map_pirq               13
 struct physdev_map_pirq {
     domid_t domid;

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/16/09 02:38, Zhang, Xiantao wrote:
> Hi, Keir/Jeremy
>    After you picking the two patches into upstream, we found it may break old dom0 when assigned a level-triggered devices to a HVM domain.  The casue is that, old dom0 can't provide trigger mode and polarity when they do map_domain_pirq.  In attached patches, they introduce a bit to indicate whether old dom0 or not.  
>
> xen-introduce-a-bit-to-identify-dom0-type.patch:  for hypervisor. 
> 0001-x86-Introduce-a-bit-MAP_COMPAT-mode-for-MAP_PIRQ_TY.patch: for pv ops dom0. 
>   

Is there any way for the dom0 kernel to tell whether the hypervisor is
implementing the new ABI, so it can choose how to set up interrupts.

MAP_COMPAT_BIT doesn't seem like a very good name, because it implies
that setting it reverts to "compatible" behaviour.  I assume that
leaving it clear enables the historical behaviour and setting it enables
the new one (since old kernels won't be setting it).

    J
> Xiantao
>
>
> Keir Fraser wrote:
>   
>> On 13/11/2009 23:57, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:
>>
>>     
>>>> Sure, I'll check it in.
>>>>
>>>>         
>>> How would you feel about backporting this into a stable release?  If
>>> we start using this interrupt setup mechanism it will be a
>>> red-letter day for dom0/xen compatibility.
>>>       
>> There probably won't be another stable release until 4.0 now. There
>> will be a 3.4.3 about the same time though.
>>
>>  -- Keir
>>     
>   

RE: Re: APIC rework

[Zhang, Xiantao]

Jeremy Fitzhardinge wrote:
> On 11/16/09 02:38, Zhang, Xiantao wrote:
>> Hi, Keir/Jeremy
>>    After you picking the two patches into upstream, we found it may
>> break old dom0 when assigned a level-triggered devices to a HVM
>> domain.  The casue is that, old dom0 can't provide trigger mode and
>> polarity when they do map_domain_pirq.  In attached patches, they
>> introduce a bit to indicate whether old dom0 or not.    
>> 
>> xen-introduce-a-bit-to-identify-dom0-type.patch:  for hypervisor.
>> 0001-x86-Introduce-a-bit-MAP_COMPAT-mode-for-MAP_PIRQ_TY.patch: for
>> pv ops dom0. 
>> 
> 
> Is there any way for the dom0 kernel to tell whether the hypervisor is
> implementing the new ABI, so it can choose how to set up interrupts.
> 
> MAP_COMPAT_BIT doesn't seem like a very good name, because it implies
> that setting it reverts to "compatible" behaviour.  I assume that
> leaving it clear enables the historical behaviour and setting it
> enables the new one (since old kernels won't be setting it).

Maybe better change to MAP_NEW_ABI_BIT ?  Since the hypervisor patch didn't change old code path after introducing this bit, so it is very easy and safe to backport to xen-3.4-testing tree, and make new dom0 be able to run top of it.  :)
Xiantao

Re: Re: APIC rework

[Keir Fraser]

On 17/11/2009 03:13, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:

>> Is there any way for the dom0 kernel to tell whether the hypervisor is
>> implementing the new ABI, so it can choose how to set up interrupts.
>> 
>> MAP_COMPAT_BIT doesn't seem like a very good name, because it implies
>> that setting it reverts to "compatible" behaviour.  I assume that
>> leaving it clear enables the historical behaviour and setting it
>> enables the new one (since old kernels won't be setting it).
> 
> Maybe better change to MAP_NEW_ABI_BIT ?  Since the hypervisor patch didn't
> change old code path after introducing this bit, so it is very easy and safe
> to backport to xen-3.4-testing tree, and make new dom0 be able to run top of
> it.  :)

It's kind of a shame to need this though. Is there no way for the hypervisor
to work out automatically whether an older dom0 is running? Or work out the
trigger/level stuff for itself (after all it parses the relevant bios tables
just like dom0)?

 -- Keir

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/16/09 19:45, Keir Fraser wrote:
> It's kind of a shame to need this though. Is there no way for the hypervisor
> to work out automatically whether an older dom0 is running? Or work out the
> trigger/level stuff for itself (after all it parses the relevant bios tables
> just like dom0)?

If Xen can set the interrupt triggering by itself, why would it ever
need dom0 to do it?  Couldn't it just preconfigure all the pins, and
then wait for dom0 to provide/request the pirq<->evtchn mapping?

    J

Re: Re: APIC rework

[Keir Fraser]

On 17/11/2009 05:20, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:

> On 11/16/09 19:45, Keir Fraser wrote:
>> It's kind of a shame to need this though. Is there no way for the hypervisor
>> to work out automatically whether an older dom0 is running? Or work out the
>> trigger/level stuff for itself (after all it parses the relevant bios tables
>> just like dom0)?
> 
> If Xen can set the interrupt triggering by itself, why would it ever
> need dom0 to do it?  Couldn't it just preconfigure all the pins, and
> then wait for dom0 to provide/request the pirq<->evtchn mapping?

Very fair question. As long as Xen knows which GSI is being addressed (which
I'm sure it can work out), the rest should follow. I don't think even things
like PCI hotplug complicate this -- IOAPIC pins themselves are static.

 -- Keir

RE: Re: APIC rework

[Zhang, Xiantao]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

Jeremy Fitzhardinge wrote:
> On 11/16/09 19:45, Keir Fraser wrote:
>> It's kind of a shame to need this though. Is there no way for the
>> hypervisor to work out automatically whether an older dom0 is
>> running? Or work out the trigger/level stuff for itself (after all
>> it parses the relevant bios tables just like dom0)?
> 
> If Xen can set the interrupt triggering by itself, why would it ever
> need dom0 to do it?  Couldn't it just preconfigure all the pins, and
> then wait for dom0 to provide/request the pirq<->evtchn mapping?


After reviewing the logic, I think we can use DOMID_SELF to identify new dom0.  In linux-2.6.18 dom0, only qemu uses this hypercall for device assginment, so map->domid shouldn't be dom0.  If old dom0/qemu with this hypercall, keeps the logic unchanged, and only change the logic for new dom0 when call into it.   Attached the patch. 
Xiantao

[-- Attachment #2: fix-compatibility-issue-about-physdev_map_pirq.patch --]
[-- Type: application/octet-stream, Size: 2197 bytes --]

x86: fix compatibility issue about physdev_map_pirq.

Keep this hypercall's logic unchanged with old dom0/qemu,
and also modify it to work with new dom0. 

Signed-off-by: Xiantao Zhang <xiantao.zhang@intel.com>


diff -r bec27eb6f72c xen/arch/x86/irq.c
--- a/xen/arch/x86/irq.c	Sat Nov 14 10:32:59 2009 +0000
+++ b/xen/arch/x86/irq.c	Tue Nov 17 17:56:46 2009 +0800
@@ -1427,13 +1427,6 @@ int map_domain_pirq(
         return -EINVAL;
     }
 
-    if ( desc->action )
-    {
-        dprintk(XENLOG_G_WARNING, "Attempt to map in-use IRQ by Xen,"
-                        " irq:%d!\n", irq);
-        return 0;
-    }
-
     old_irq = domain_pirq_to_irq(d, pirq);
     old_pirq = domain_irq_to_pirq(d, irq);
 
@@ -1452,7 +1445,6 @@ int map_domain_pirq(
                 d->domain_id, pirq);
         return ret;
     }
-
 
     if ( type == MAP_PIRQ_TYPE_MSI )
     {
diff -r bec27eb6f72c xen/arch/x86/physdev.c
--- a/xen/arch/x86/physdev.c	Sat Nov 14 10:32:59 2009 +0000
+++ b/xen/arch/x86/physdev.c	Tue Nov 17 19:00:09 2009 +0800
@@ -59,10 +59,6 @@ static int physdev_map_pirq(struct physd
             int gsi, triggering, polarity;
             
             gsi = map->index & 0xffff;
-            triggering = !!(map->index & (1 << 16));
-            polarity = !!(map->index & (1 << 24));
-            irq = pirq = map->pirq;
-            
             if ( gsi < 0 || gsi >= nr_irqs_gsi )
             {
                 dprintk(XENLOG_G_ERR, "dom%d: map invalid gsi %d\n",
@@ -70,7 +66,15 @@ static int physdev_map_pirq(struct physd
                 ret = -EINVAL;
                 goto free_domain;
             }
+            if (map->domid == DOMID_SELF)
+                irq = pirq = map->pirq;
+            else {
+                irq = domain_pirq_to_irq(current->domain, map->pirq);
+                pirq = domain_irq_to_pirq(d, irq);
+            }
             if ( !check_irq_status(irq) ) {
+                triggering = !!(map->index & (1 << 16));
+                polarity = !!(map->index & (1 << 24));
                 mp_register_gsi(gsi, triggering, polarity);
                 printk("Register gsi:%d for dom:%d, irq:%d\n", gsi,
                       d->domain_id, irq);

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: APIC rework

[Keir Fraser]

On 17/11/2009 12:46, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:

>> If Xen can set the interrupt triggering by itself, why would it ever
>> need dom0 to do it?  Couldn't it just preconfigure all the pins, and
>> then wait for dom0 to provide/request the pirq<->evtchn mapping?
> 
> After reviewing the logic, I think we can use DOMID_SELF to identify new dom0.
> In linux-2.6.18 dom0, only qemu uses this hypercall for device assginment, so
> map->domid shouldn't be dom0.  If old dom0/qemu with this hypercall, keeps the
> logic unchanged, and only change the logic for new dom0 when call into it.
> Attached the patch.

Don't like it (subtle semantic difference based on domid field is very odd
and could bite us later), and actually now I look closer I don't like the
original patch much either, if only because it does clearly change the
interface for MAP_PIRQ_TYPE_GSI by adding trigger/polarity flags (and not
documented or exported properly in the physdev.h header file).

I need to take a step back and work out what in fact you're trying to
achieve. I'm going to revert the original patch from xen-unstable. If such
an interface extension is really required, I think at least the new
interface should be a new subtype of MAP_PIRQ_TYPE_xxx, properly described
in the header file. But like I say, I don't know what the patch is even
really trying to do or overcome.

 -- Keir

RE: Re: APIC rework

[Zhang, Xiantao]

Keir Fraser wrote:
> On 17/11/2009 12:46, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:
> 
>>> If Xen can set the interrupt triggering by itself, why would it ever
>>> need dom0 to do it?  Couldn't it just preconfigure all the pins, and
>>> then wait for dom0 to provide/request the pirq<->evtchn mapping?
>> 
>> After reviewing the logic, I think we can use DOMID_SELF to identify
>> new dom0. In linux-2.6.18 dom0, only qemu uses this hypercall for
>> device assginment, so map->domid shouldn't be dom0.  If old
>> dom0/qemu with this hypercall, keeps the logic unchanged, and only
>> change the logic for new dom0 when call into it. Attached the patch.
> 
> Don't like it (subtle semantic difference based on domid field is
> very odd and could bite us later), and actually now I look closer I
> don't like the original patch much either, if only because it does
> clearly change the interface for MAP_PIRQ_TYPE_GSI by adding
> trigger/polarity flags (and not documented or exported properly in
> the physdev.h header file). 
> 
> I need to take a step back and work out what in fact you're trying to
> achieve. I'm going to revert the original patch from xen-unstable. If
> such an interface extension is really required, I think at least the
> new interface should be a new subtype of MAP_PIRQ_TYPE_xxx, properly
> described in the header file. But like I say, I don't know what the
> patch is even really trying to do or overcome. 

Originally, this patch is target to get rid of ioapic changes in dom0. Before this patch, GSI irq should be mapped and setup through dom0 programming ioapic entries, but it depends on using ioapic logic in dom0. And if we remove ioapic logic from dom0,  we need to find new way how to setup GSI irq.  And this patch comes out under this situation.  The idea is from that in Xen the interface MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for MSI IRQ for each domain. Since MSI IRQ can be setup through this hypercall, and I think  we also can leverage the interface MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq. Further analysis showes that this interface is only used for assigning devices to HVM domain in qemu, and I think it should be Okay for dom0 building the mapping between its pirq and irq. One different thing for GSI irq is that more info should be provided in the call, since GSI IRQ has different trigger-mode and polarity (originally it is provided by ioapic write in dom0). Certainly, I also think we need to document the related info, and if you agree to the change, I am happy to add it. 
Xiantao

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/17/09 06:17, Zhang, Xiantao wrote:
> Originally, this patch is target to get rid of ioapic changes in dom0. Before this patch, GSI irq should be mapped and setup through dom0 programming ioapic entries, but it depends on using ioapic logic in dom0. And if we remove ioapic logic from dom0,  we need to find new way how to setup GSI irq.  And this patch comes out under this situation.  The idea is from that in Xen the interface MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for MSI IRQ for each domain. Since MSI IRQ can be setup through this hypercall, and I think  we also can leverage the interface MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq. Further analysis showes that this interface is only used for assigning devices to HVM domain in qemu, and I think it should be Okay for dom0 building the mapping between its pirq and irq. One different thing for GSI irq is that more info should be provided in the call, since GSI IRQ has different trigger-mode and polarity (originally it is provided by ioapic write in dom0). Certainly, I also think we need to document the related info, and if you agree to the change, I am happy to add it. 
>   

I don't think there's any need to overload the existing interface
though.  If we're adding new functionality then we can add a new
interface for it (but with luck we can reuse most of the existing code
to implement it).

If you're already considering a "treat this differently" flag in the
argument, then that's a strong pointer that a new interface is warranted.

But also consider the question whether Xen needs to get triggering info
from dom0 at all, or whether it can correctly derive that info from its
own parsing of the relevant tables.

    J

RE: Re: APIC rework

[Zhang, Xiantao]

Jeremy Fitzhardinge wrote:
> On 11/17/09 06:17, Zhang, Xiantao wrote:
>> Originally, this patch is target to get rid of ioapic changes in
>> dom0. Before this patch, GSI irq should be mapped and setup through
>> dom0 programming ioapic entries, but it depends on using ioapic
>> logic in dom0. And if we remove ioapic logic from dom0,  we need to
>> find new way how to setup GSI irq.  And this patch comes out under
>> this situation.  The idea is from that in Xen the interface
>> MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for MSI
>> IRQ for each domain. Since MSI IRQ can be setup through this
>> hypercall, and I think  we also can leverage the interface
>> MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq. Further
>> analysis showes that this interface is only used for assigning
>> devices to HVM domain in qemu, and I think it should be Okay for
>> dom0 building the mapping between its pirq and irq. One different
>> thing for GSI irq is that more info should be provided in the call,
>> since GSI IRQ has different trigger-mode and polarity (originally it
>> is provided by ioapic write in dom0). Certainly, I also think we
>> need to document the related info, and if you agree to the change, I
>> am happy to add it.                 
>> 
> 
> I don't think there's any need to overload the existing interface
> though.  If we're adding new functionality then we can add a new
> interface for it (but with luck we can reuse most of the existing code
> to implement it).

> If you're already considering a "treat this differently" flag in the
> argument, then that's a strong pointer that a new interface is
> warranted. 

Agree, and I also don't object to add a similar interface.  
Since this existing interface is only used for hvm domain before, and  just want to re-use it for dom. 


> But also consider the question whether Xen needs to get triggering
> info from dom0 at all, or whether it can correctly derive that info
> from its own parsing of the relevant tables.

Xen should have the ability to parse the info, but may need more logic porting to hyervisor. 

Xiantao

Re: Re: APIC rework

[Konrad Rzeszutek Wilk]

On Wed, Nov 18, 2009 at 11:37:51AM +0800, Zhang, Xiantao wrote:
> Jeremy Fitzhardinge wrote:
> > On 11/17/09 06:17, Zhang, Xiantao wrote:
> >> Originally, this patch is target to get rid of ioapic changes in
> >> dom0. Before this patch, GSI irq should be mapped and setup through
> >> dom0 programming ioapic entries, but it depends on using ioapic
> >> logic in dom0. And if we remove ioapic logic from dom0,  we need to
> >> find new way how to setup GSI irq.  And this patch comes out under
> >> this situation.  The idea is from that in Xen the interface
> >> MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for MSI
> >> IRQ for each domain. Since MSI IRQ can be setup through this
> >> hypercall, and I think  we also can leverage the interface
> >> MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq. Further
> >> analysis showes that this interface is only used for assigning
> >> devices to HVM domain in qemu, and I think it should be Okay for
> >> dom0 building the mapping between its pirq and irq. One different
> >> thing for GSI irq is that more info should be provided in the call,
> >> since GSI IRQ has different trigger-mode and polarity (originally it
> >> is provided by ioapic write in dom0). Certainly, I also think we
> >> need to document the related info, and if you agree to the change, I
> >> am happy to add it.                 
> >> 
> > 
> > I don't think there's any need to overload the existing interface
> > though.  If we're adding new functionality then we can add a new
> > interface for it (but with luck we can reuse most of the existing code
> > to implement it).
> 
> > If you're already considering a "treat this differently" flag in the
> > argument, then that's a strong pointer that a new interface is
> > warranted. 
> 
> Agree, and I also don't object to add a similar interface.  
> Since this existing interface is only used for hvm domain before, and  just want to re-use it for dom. 

I am pretty sure it is used for PV domains too. Look in 'xen_create_msi_irq', which
extracts the domain that has a PCI device for pass-through and on behalf of that
domain makes the MAP_PIRQ_TYPE_MSI call.

Furthermore, it is also used by Dom0 for MSI devices.

What do you mean by 're-use it for dom'?

RE: Re: APIC rework

[Zhang, Xiantao]

Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 18, 2009 at 11:37:51AM +0800, Zhang, Xiantao wrote:
>> Jeremy Fitzhardinge wrote:
>>> On 11/17/09 06:17, Zhang, Xiantao wrote:
>>>> Originally, this patch is target to get rid of ioapic changes in
>>>> dom0. Before this patch, GSI irq should be mapped and setup through
>>>> dom0 programming ioapic entries, but it depends on using ioapic
>>>> logic in dom0. And if we remove ioapic logic from dom0,  we need to
>>>> find new way how to setup GSI irq.  And this patch comes out under
>>>> this situation.  The idea is from that in Xen the interface
>>>> MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for MSI
>>>> IRQ for each domain. Since MSI IRQ can be setup through this
>>>> hypercall, and I think  we also can leverage the interface
>>>> MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq. Further
>>>> analysis showes that this interface is only used for assigning
>>>> devices to HVM domain in qemu, and I think it should be Okay for
>>>> dom0 building the mapping between its pirq and irq. One different
>>>> thing for GSI irq is that more info should be provided in the call,
>>>> since GSI IRQ has different trigger-mode and polarity (originally
>>>> it is provided by ioapic write in dom0). Certainly, I also think we
>>>> need to document the related info, and if you agree to the change,
>>>> I am happy to add it. 
>>>> 
>>> 
>>> I don't think there's any need to overload the existing interface
>>> though.  If we're adding new functionality then we can add a new
>>> interface for it (but with luck we can reuse most of the existing
>>> code to implement it).
>> 
>>> If you're already considering a "treat this differently" flag in the
>>> argument, then that's a strong pointer that a new interface is
>>> warranted.
>> 
>> Agree, and I also don't object to add a similar interface.
>> Since this existing interface is only used for hvm domain before,
>> and  just want to re-use it for dom. 
> 
> I am pretty sure it is used for PV domains too. Look in
> 'xen_create_msi_irq', which extracts the domain that has a PCI device
> for pass-through and on behalf of that domain makes the
> MAP_PIRQ_TYPE_MSI call. 
> 
> Furthermore, it is also used by Dom0 for MSI devices.

We mean the call MAP_PIRQ_TYPE_GSI, not MAP_PIRQ_TYPE_MSI. 

> What do you mean by 're-use it for dom'?
Oh, seems the char "0" is missing, should be dom0. 
Xiantao

Re: Re: APIC rework

[Keir Fraser]

Xiantao,

Responding inline below with stupid/basic questions. Just want to go back to
first principles with this and work out which of us is on the wrong track.

On 17/11/2009 14:17, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:

> Originally, this patch is target to get rid of ioapic changes in dom0. Before
> this patch, GSI irq should be mapped and setup through dom0 programming ioapic
> entries, but it depends on using ioapic logic in dom0.

So dom0 doesn't write the IOAPIC *at all*? Okay.

> And if we remove ioapic
> logic from dom0,  we need to find new way how to setup GSI irq.  And this
> patch comes out under this situation.

Why does this need to be done under dom0 control? All GSIs are parseable by
Xen by itself aren't they, from MPBIOS tables or ACPI MADT? So at least Xen
should be able to work out for itself APIC pin -> GSI mappings, and pol/trig
settings.

> The idea is from that in Xen the
> interface MAP_PIRQ_TYPE_MSI is used to build the pirq and irq mapping for MSI
> IRQ for each domain. Since MSI IRQ can be setup through this hypercall, and I
> think  we also can leverage the interface MAP_PIRQ_TYPE _GSI to build the
> mapping for GSI irq.

For modern dom0 don't we already assume that dom0 pirq == irq == gsi (see
comments in ioapic_guest_write)? Perhaps we should just have that
relationship set up by default: I think only NetBSD dom0 has different, and
it will establish different relationship via legacy method of
PHYSDEVOP_alloc_irq_vector and paravirtualised IOAPIC writes?

> Further analysis showes that this interface is only used
> for assigning devices to HVM domain in qemu, and I think it should be Okay for
> dom0 building the mapping between its pirq and irq. One different thing for
> GSI irq is that more info should be provided in the call, since GSI IRQ has
> different trigger-mode and polarity (originally it is provided by ioapic write
> in dom0). Certainly, I also think we need to document the related info, and if
> you agree to the change, I am happy to add it.

Maybe this follows if my above assumptions/arguments are broken.

 -- Keir

RE: Re: APIC rework

[Jiang, Yunhong]

>> And if we remove ioapic
>> logic from dom0,  we need to find new way how to setup GSI irq.  And
>> this patch comes out under this situation.
> 
> Why does this need to be done under dom0 control? All GSIs are
> parseable by Xen by itself aren't they, from MPBIOS tables or ACPI
> MADT? So 
> at least Xen
> should be able to work out for itself APIC pin -> GSI
> mappings, and pol/trig
> settings.

My 2 cents for this topic, although I'm still trying to figure out the whole picture of the patch and discussion thread.

The ACPI MADT table gives the relationship between IOAPIC and gsi, while DSDT table's _PRT gives the relationship between PCI devices and GSI.

Two way provided in ACPI _PRT table. If the source filed in _PRT entry does not refert to any device, it means a GSI. I didn't find any hint in ACPI spec on the polarity/level should be configured. Currently kernel assume the polarity/level is fixed as low/level, which I suspect is according to PCI spec.

If the _PRT table present the interrupt as PCI Interrupt Link Device, that means the interrupt attribute is configurable, OSPM need figure out the polarity/level information through _CRS/_PRS method in these objectes.

For method 1 (i.e. source filed does not refer to device), maybe we can assume the attribute is fixed and Keir's suggestion will work, while I suspect if Xen can do anyting for second type.

Quickly check Xiantao's patch, I suspect if it will work for 2nd situation. 

BTW, I don't know know any system which use 2nd type when working in APIC mode.

--jyh

RE: Re: APIC rework

[Zhang, Xiantao]

Keir Fraser wrote:
> Xiantao,
> 
> Responding inline below with stupid/basic questions. Just want to go
> back to first principles with this and work out which of us is on the
> wrong track. 
> 
> On 17/11/2009 14:17, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:
> 
>> Originally, this patch is target to get rid of ioapic changes in
>> dom0. Before this patch, GSI irq should be mapped and setup through
>> dom0 programming ioapic entries, but it depends on using ioapic
>> logic in dom0. 
> 
> So dom0 doesn't write the IOAPIC *at all*? Okay.

Dom0 won't write ioapic in new dom0, and all operations related to ioapic should be dummy, won't trap to hypervisor any more. 

>> And if we remove ioapic
>> logic from dom0,  we need to find new way how to setup GSI irq.  And
>> this patch comes out under this situation.
> 
> Why does this need to be done under dom0 control? All GSIs are
> parseable by Xen by itself aren't they, from MPBIOS tables or ACPI
> MADT? So at least Xen should be able to work out for itself APIC pin
> -> GSI mappings, and pol/trig settings.

I agree with you at this point, and Xen should has the ability to parse the info, but current Xen should doesn't have.
A clean way is to do this in hyervisor and dom0 only ask the request about GSI IRQ, like MSI does today. 

>> The idea is from that in Xen the
>> interface MAP_PIRQ_TYPE_MSI is used to build the pirq and irq
>> mapping for MSI IRQ for each domain. Since MSI IRQ can be setup
>> through this hypercall, and I think  we also can leverage the
>> interface MAP_PIRQ_TYPE _GSI to build the mapping for GSI irq.
> 
> For modern dom0 don't we already assume that dom0 pirq == irq == gsi
> (see comments in ioapic_guest_write)? Perhaps we should just have that
> relationship set up by default: I think only NetBSD dom0 has
> different, and it will establish different relationship via legacy
> method of PHYSDEVOP_alloc_irq_vector and paravirtualised IOAPIC
> writes?

The assumption should be right for dom0 today, but it still needs to register this info to dom0's private data(d->arch.{pirq_irq, irq_pirq).
And I think maybe we should clean up the logic and let hypervisor knows the assumption, when consulting this relationship.

Xiantao

Re: Re: APIC rework

[Keir Fraser]

On 18/11/2009 03:25, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:

>> For modern dom0 don't we already assume that dom0 pirq == irq == gsi
>> (see comments in ioapic_guest_write)? Perhaps we should just have that
>> relationship set up by default: I think only NetBSD dom0 has
>> different, and it will establish different relationship via legacy
>> method of PHYSDEVOP_alloc_irq_vector and paravirtualised IOAPIC
>> writes?
> 
> The assumption should be right for dom0 today, but it still needs to register
> this info to dom0's private data(d->arch.{pirq_irq, irq_pirq).
> And I think maybe we should clean up the logic and let hypervisor knows the
> assumption, when consulting this relationship.

Well, it strikes me that existing MAP_PIRQ_TYPE_GSI fills this role already,
as it is, doesn't it? Seems to me that is its whole purpose. :-)

Shoehorning trig/pol information into it as well is kind of nasty. And I
think on any PC system it should suffice to assume GSI 0-15 are ISA
edge-triggered active-high, GSI 16+ are PCI level-triggered active-low, and
any exceptions are parsed out of MADT or MPBIOS. We pretty much have all
that code, it just might need plumbing back in a little bit. Yunhong points
out that ACPI DSDT can have overriding objects in the _PRT, but I don't know
it ever actually gets used on real-world PC systems. So I would try without,
but if we do end up needing to get this info from dom0, I think it should be
via a new physdev_op.

 -- Keir

RE: Re: APIC rework

[Zhang, Xiantao]

[-- Attachment #1: Type: text/plain, Size: 2233 bytes --]

Keir Fraser wrote:
> On 18/11/2009 03:25, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:
> 
>>> For modern dom0 don't we already assume that dom0 pirq == irq == gsi
>>> (see comments in ioapic_guest_write)? Perhaps we should just have
>>> that relationship set up by default: I think only NetBSD dom0 has
>>> different, and it will establish different relationship via legacy
>>> method of PHYSDEVOP_alloc_irq_vector and paravirtualised IOAPIC
>>> writes?
>> 
>> The assumption should be right for dom0 today, but it still needs to
>> register this info to dom0's private data(d->arch.{pirq_irq,
>> irq_pirq). 
>> And I think maybe we should clean up the logic and let hypervisor
>> knows the assumption, when consulting this relationship.
> 
> Well, it strikes me that existing MAP_PIRQ_TYPE_GSI fills this role
> already, as it is, doesn't it? Seems to me that is its whole purpose.
> :-) 
> 
> Shoehorning trig/pol information into it as well is kind of nasty.
> And I think on any PC system it should suffice to assume GSI 0-15 are
> ISA edge-triggered active-high, GSI 16+ are PCI level-triggered
> active-low, and any exceptions are parsed out of MADT or MPBIOS. We
> pretty much have all that code, it just might need plumbing back in a
> little bit. Yunhong points out that ACPI DSDT can have overriding
> objects in the _PRT, but I don't know it ever actually gets used on
> real-world PC systems. So I would try without, but if we do end up
> needing to get this info from dom0, I think it should be via a new
> physdev_op.

At least dom0 parses this info from DSDT, so we can't have the assuption whether it is used or not, I think. And I also agree to add a new physdev_op to handle this case, and it should be better way to go.  
Based on this idea, I worked out the patch, attached!  In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each domain can require to map each GSI in this case. 
In addition, I believe it is very safe to port the hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed.  BTW, I also tested apic and non-apic cases, they works fine after applying the patches. 
Xiantao



[-- Attachment #2: x86-add-a-new-physdev_op.patch --]
[-- Type: application/octet-stream, Size: 4250 bytes --]

x86: Add a new physdev_op PHYSDEVOP_setup_gsi for GSI setup.

GSI 0-15 is setup by hypervisor, and GSI > =16 is setup by dom0
this physdev_op PHYSDEVOP_setup_gsi. This patch can help dom0
to get rid of intrusive changes of ioapic.

Signed-off-by: Xiantao Zhang <xiantao.zhang@intel.com>

diff -r 2864f1942790 xen/arch/x86/mpparse.c
--- a/xen/arch/x86/mpparse.c	Mon Nov 23 07:23:07 2009 +0000
+++ b/xen/arch/x86/mpparse.c	Mon Nov 23 21:46:24 2009 +0800
@@ -1122,7 +1122,7 @@ int mp_register_gsi (u32 gsi, int trigge
 	ioapic = mp_find_ioapic(gsi);
 	if (ioapic < 0) {
 		printk(KERN_WARNING "No IOAPIC for GSI %u\n", gsi);
-		return gsi;
+		return -EINVAL;
 	}
 
 	ioapic_pin = gsi - mp_ioapic_routing[ioapic].gsi_base;
@@ -1141,12 +1141,12 @@ int mp_register_gsi (u32 gsi, int trigge
 		printk(KERN_ERR "Invalid reference to IOAPIC pin "
 			"%d-%d\n", mp_ioapic_routing[ioapic].apic_id, 
 			ioapic_pin);
-		return gsi;
+		return -EINVAL;
 	}
 	if ((1<<bit) & mp_ioapic_routing[ioapic].pin_programmed[idx]) {
 		Dprintk(KERN_DEBUG "Pin %d-%d already programmed\n",
 			mp_ioapic_routing[ioapic].apic_id, ioapic_pin);
-		return gsi_to_irq[gsi];
+		return -EEXIST;
 	}
 
 	mp_ioapic_routing[ioapic].pin_programmed[idx] |= (1<<bit);
@@ -1180,14 +1180,13 @@ int mp_register_gsi (u32 gsi, int trigge
 			gsi_to_irq[irq] = gsi;
 		} else {
 			printk(KERN_ERR "GSI %u is too high\n", gsi);
-			return gsi;
+			return -E2BIG;
 		}
 	}
 
-	io_apic_set_pci_routing(ioapic, ioapic_pin, gsi,
+	return io_apic_set_pci_routing(ioapic, ioapic_pin, gsi,
 		    triggering == ACPI_EDGE_SENSITIVE ? 0 : 1,
 		    polarity == ACPI_ACTIVE_HIGH ? 0 : 1);
-	return gsi;
 }
 
 #endif /* CONFIG_X86_IO_APIC */
diff -r 2864f1942790 xen/arch/x86/physdev.c
--- a/xen/arch/x86/physdev.c	Mon Nov 23 07:23:07 2009 +0000
+++ b/xen/arch/x86/physdev.c	Tue Nov 24 15:47:57 2009 +0800
@@ -62,13 +62,18 @@ static int physdev_map_pirq(struct physd
                 ret = -EINVAL;
                 goto free_domain;
             }
+
             irq = domain_pirq_to_irq(current->domain, map->index);
             if ( !irq )
             {
-                dprintk(XENLOG_G_ERR, "dom%d: map pirq with incorrect irq!\n",
-                        d->domain_id);
-                ret = -EINVAL;
-                goto free_domain;
+                if ( IS_PRIV(current->domain) )
+                    irq = map->index;
+                else {
+                    dprintk(XENLOG_G_ERR, "dom%d: map pirq with incorrect irq!\n",
+                            d->domain_id);
+                    ret = -EINVAL;
+                    goto free_domain;
+                }
             }
             break;
 
@@ -457,6 +462,28 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_H
         spin_unlock(&pcidevs_lock);
         break;
     }
+    case PHYSDEVOP_setup_gsi: {
+        struct physdev_setup_gsi setup_gsi;
+
+        ret = -EPERM;
+        if ( !IS_PRIV(v->domain) )
+            break;
+
+        ret = -EFAULT;
+        if ( copy_from_guest(&setup_gsi, arg, 1) != 0 )
+            break;
+        
+        ret = -EINVAL;
+        if ( setup_gsi.gsi < 0 || setup_gsi.gsi >= nr_irqs_gsi )
+            break;
+        /* GSI < 16 has been setup by hypervisor */
+        if ( setup_gsi.gsi >= 16 )
+            ret = mp_register_gsi(setup_gsi.gsi, setup_gsi.triggering,
+                            setup_gsi.polarity);
+        else 
+            ret = -EEXIST;
+        break; 
+    }
     default:
         ret = -ENOSYS;
         break;
diff -r 2864f1942790 xen/include/public/physdev.h
--- a/xen/include/public/physdev.h	Mon Nov 23 07:23:07 2009 +0000
+++ b/xen/include/public/physdev.h	Mon Nov 23 18:12:44 2009 +0800
@@ -227,6 +227,19 @@ typedef struct physdev_op physdev_op_t;
 typedef struct physdev_op physdev_op_t;
 DEFINE_XEN_GUEST_HANDLE(physdev_op_t);
 
+#define PHYSDEVOP_setup_gsi    21
+struct physdev_setup_gsi {
+    int gsi;
+    /* IN */
+    uint8_t triggering;
+    /* IN */
+    uint8_t polarity;
+    /* IN */
+};
+
+typedef struct physdev_setup_gsi physdev_setup_gsi_t;
+DEFINE_XEN_GUEST_HANDLE(physdev_setup_gsi_t);
+
 /*
  * Notify that some PIRQ-bound event channels have been unmasked.
  * ** This command is obsolete since interface version 0x00030202 and is **

[-- Attachment #3: 0001-x86-ioapic-Remove-Xen-s-specific-changes-about-ioa.patch --]
[-- Type: application/octet-stream, Size: 11439 bytes --]

From bdf93646f4b7008d364944bd3e3cef67e043e2c1 Mon Sep 17 00:00:00 2001
From: Xiantao Zhang <xiantao.zhang@intel.com>
Date: Tue, 24 Nov 2009 15:26:47 -0500
Subject: [PATCH] x86: ioapic: Remove Xen's specific changes about ioapic logic.

Remove the intrusive changes to ioapic logic for Xen.

Signed-off-by: Xiantao Zhang <xiantao.zhang@intel.com>
---
 arch/x86/include/asm/io_apic.h  |    4 --
 arch/x86/kernel/acpi/boot.c     |    3 ++
 arch/x86/kernel/apic/io_apic.c  |   21 -------------
 arch/x86/xen/apic.c             |   25 ---------------
 arch/x86/xen/mmu.c              |    4 ++-
 arch/x86/xen/pci.c              |   63 ++++++++++++++++++--------------------
 drivers/xen/events.c            |   46 +++++-----------------------
 include/xen/interface/physdev.h |   10 ++++++
 9 files changed, 57 insertions(+), 125 deletions(-)

diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
index 70f5ea9..b45dedf 100644
--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -188,9 +188,5 @@ static inline void probe_nr_irqs_gsi(void)	{ }
 #endif
 
 void xen_io_apic_init(void);
-unsigned int xen_io_apic_read(unsigned apic, unsigned reg);
-void xen_io_apic_write(unsigned int apic,
-		       unsigned int reg, unsigned int value);
-
 
 #endif /* _ASM_X86_IO_APIC_H */
diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
index d47c54f..28f83dd 100644
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -969,6 +969,9 @@ int __init acpi_probe_gsi(void)
 			max_gsi = gsi;
 	}
 
+	if (xen_initial_domain())
+		max_gsi += 255; /* Plus maximum entries of an ioapic. */
+
 	return max_gsi + 1;
 }
 
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index b9a0e67..9764b1a 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -386,9 +386,6 @@ static inline unsigned int io_apic_read(unsigned int apic, unsigned int reg)
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain())
-		return xen_io_apic_read(apic, reg);
-
 	io_apic = io_apic_base(apic);
 	writel(reg, &io_apic->index);
 	return readl(&io_apic->data);
@@ -398,11 +395,6 @@ static inline void io_apic_write(unsigned int apic, unsigned int reg, unsigned i
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_write(apic, reg, value);
-		return;
-	}
-
 	io_apic = io_apic_base(apic);
 	writel(reg, &io_apic->index);
 	writel(value, &io_apic->data);
@@ -418,11 +410,6 @@ static inline void io_apic_modify(unsigned int apic, unsigned int reg, unsigned
 {
 	struct io_apic __iomem *io_apic;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_write(apic, reg, value);
-		return;
-	}
-
 	io_apic = io_apic_base(apic);
 
 	if (sis_apic_bug)
@@ -3162,9 +3149,6 @@ static int __init ioapic_init_sysfs(void)
 	struct sys_device * dev;
 	int i, size, error;
 
-	if (xen_initial_domain())
-		return 0;
-
 	error = sysdev_class_register(&ioapic_sysdev_class);
 	if (error)
 		return error;
@@ -4202,11 +4186,6 @@ void __init ioapic_init_mappings(void)
 	struct resource *ioapic_res;
 	int i;
 
-	if (xen_initial_domain()) {
-		xen_io_apic_init();
-		return;
-	}
-
 	ioapic_res = ioapic_setup_resources();
 	for (i = 0; i < nr_ioapics; i++) {
 		if (smp_found_config) {
diff --git a/arch/x86/xen/apic.c b/arch/x86/xen/apic.c
index ee0db39..805ce70 100644
--- a/arch/x86/xen/apic.c
+++ b/arch/x86/xen/apic.c
@@ -17,31 +17,6 @@ void __init xen_io_apic_init(void)
 	enable_IO_APIC();
 }
 
-unsigned int xen_io_apic_read(unsigned apic, unsigned reg)
-{
-	struct physdev_apic apic_op;
-	int ret;
-
-	apic_op.apic_physbase = mp_ioapics[apic].apicaddr;
-	apic_op.reg = reg;
-	ret = HYPERVISOR_physdev_op(PHYSDEVOP_apic_read, &apic_op);
-	if (ret)
-		BUG();
-	return apic_op.value;
-}
-
-
-void xen_io_apic_write(unsigned int apic, unsigned int reg, unsigned int value)
-{
-	struct physdev_apic apic_op;
-
-	apic_op.apic_physbase = mp_ioapics[apic].apicaddr;
-	apic_op.reg = reg;
-	apic_op.value = value;
-	if (HYPERVISOR_physdev_op(PHYSDEVOP_apic_write, &apic_op))
-		BUG();
-}
-
 void xen_init_apic(void)
 {
 	if (!xen_initial_domain())
diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index c5e31cb..8095df6 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1938,6 +1938,8 @@ __init pgd_t *xen_setup_kernel_pagetable(pgd_t *pgd,
 }
 #endif	/* CONFIG_X86_64 */
 
+static unsigned char dummy_ioapic_mapping[PAGE_SIZE] __page_aligned_bss;
+
 static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)
 {
 	pte_t pte;
@@ -1976,7 +1978,7 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)
 		 * We just don't map the IO APIC - all access is via
 		 * hypercalls.  Keep the address in the pte for reference.
 		 */
-		pte = pfn_pte(phys, PAGE_NONE);
+		pte = __pte(__pa(dummy_ioapic_mapping) | __PAGE_KERNEL);
 		break;
 #endif
 	case FIX_PARAVIRT_BOOTMAP:
diff --git a/arch/x86/xen/pci.c b/arch/x86/xen/pci.c
index fc3508c..ac7e6bd 100644
--- a/arch/x86/xen/pci.c
+++ b/arch/x86/xen/pci.c
@@ -15,35 +15,11 @@
 
 #include "xen-ops.h"
 
-static void xen_set_io_apic_routing(int irq, int trigger, int polarity)
-{
-	int ioapic, ioapic_pin;
-	int vector, gsi;
-	struct IO_APIC_route_entry entry;
-
-	gsi = xen_gsi_from_irq(irq);
-	vector = xen_vector_from_irq(irq);
-
-	ioapic = mp_find_ioapic(gsi);
-	if (ioapic == -1) {
-		printk(KERN_WARNING "xen_set_ioapic_routing: irq %d gsi %d ioapic %d\n",
-			irq, gsi, ioapic);
-		return;
-	}
-
-	ioapic_pin = mp_find_ioapic_pin(ioapic, gsi);
-
-	printk(KERN_INFO "xen_set_ioapic_routing: irq %d gsi %d vector %d ioapic %d pin %d triggering %d polarity %d\n",
-		irq, gsi, vector, ioapic, ioapic_pin, trigger, polarity);
-
-	setup_ioapic_entry(ioapic, -1, &entry, ~0, trigger, polarity, vector,
-			   ioapic_pin);
-	ioapic_write_entry(ioapic, ioapic_pin, entry);
-}
-
 int xen_register_gsi(u32 gsi, int triggering, int polarity)
 {
-	int irq;
+	int rc, irq;
+	struct physdev_setup_gsi setup_gsi;
+	struct physdev_map_pirq map_irq;
 	int shareable = 0;
 	char *name;
 
@@ -51,7 +27,7 @@ int xen_register_gsi(u32 gsi, int triggering, int polarity)
 		return -1;
 
 	printk(KERN_DEBUG "xen: registering gsi %u triggering %d polarity %d\n",
-	       gsi, triggering, polarity);
+			gsi, triggering, polarity);
 
 	if (triggering == ACPI_EDGE_SENSITIVE) {
 		shareable = 0;
@@ -65,11 +41,32 @@ int xen_register_gsi(u32 gsi, int triggering, int polarity)
 
 	printk(KERN_DEBUG "xen: --> irq=%d\n", irq);
 
-	if (irq >= 0)
-		xen_set_io_apic_routing(irq,
-					triggering == ACPI_EDGE_SENSITIVE ? 0 : 1,
-					polarity == ACPI_ACTIVE_HIGH ? 0 : 1);
-
+	if (irq >= 0) {
+		setup_gsi.gsi = gsi;
+		setup_gsi.triggering = (triggering == ACPI_EDGE_SENSITIVE ?
+				0 : 1);
+		setup_gsi.polarity = (polarity == ACPI_ACTIVE_HIGH ? 0 : 1);
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_setup_gsi, &setup_gsi);
+		if (rc == -EEXIST)
+			printk(KERN_INFO "Already setup the GSI :%d\n", gsi);
+		else if (rc) {
+			printk(KERN_ERR "Failed to setup GSI :%d, err_code:%d\n",
+					gsi, rc);
+			BUG();
+		}
+
+		map_irq.domid = DOMID_SELF;
+		map_irq.type = MAP_PIRQ_TYPE_GSI;
+		map_irq.index = gsi;
+		map_irq.pirq = irq;
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);
+		if (rc) {
+			printk(KERN_WARNING "xen map irq failed %d\n", rc);
+			irq = -1;
+		}
+	}
 	return irq;
 }
 
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index adc8c09..d990d11 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -75,7 +75,7 @@ enum xen_irq_type {
  * event channel - irq->event channel mapping
  * cpu - cpu this event channel is bound to
  * index - type-specific information:
- *    PIRQ - vector, with MSB being "needs EIO"
+ *    PIRQ - with MSB being "needs EIO"
  *    VIRQ - virq number
  *    IPI - IPI vector
  *    EVTCHN -
@@ -91,7 +91,6 @@ struct irq_info
 		enum ipi_vector ipi;
 		struct {
 			unsigned short nr;
-			unsigned char vector;
 			unsigned char flags;
 			domid_t domid;
 		} pirq;
@@ -148,11 +147,11 @@ static struct irq_info mk_virq_info(unsigned short evtchn, unsigned short virq)
 }
 
 static struct irq_info mk_pirq_info(unsigned short evtchn,
-				    unsigned short pirq, unsigned short vector)
+				    unsigned short pirq)
 {
 	return (struct irq_info) { .type = IRQT_PIRQ, .evtchn = evtchn,
 			.cpu = 0, .u.pirq =
-			{ .nr = pirq, .vector = vector, .domid = DOMID_SELF } };
+			{ .nr = pirq, .domid = DOMID_SELF } };
 }
 
 /*
@@ -204,16 +203,6 @@ static unsigned gsi_from_irq(unsigned irq)
 	return info->u.pirq.nr;
 }
 
-static unsigned vector_from_irq(unsigned irq)
-{
-	struct irq_info *info = info_for_irq(irq);
-
-	BUG_ON(info == NULL);
-	BUG_ON(info->type != IRQT_PIRQ);
-
-	return info->u.pirq.vector;
-}
-
 static enum xen_irq_type type_from_irq(unsigned irq)
 {
 	return info_for_irq(irq)->type;
@@ -449,7 +438,7 @@ static unsigned int startup_pirq(unsigned int irq)
 	if (rc != 0) {
 		if (!probing_irq(irq))
 			printk(KERN_INFO "Failed to obtain physical IRQ %d\n",
-			       irq);
+				irq);
 		return 0;
 	}
 	evtchn = bind_pirq.port;
@@ -545,14 +534,13 @@ static int find_irq_by_gsi(unsigned gsi)
 }
 
 /*
- * Allocate a physical irq, along with a vector.  We don't assign an
- * event channel until the irq actually started up.  Return an
+ * Allocate a physical irq.  We don't assign an event channel
+ * until the irq actually started up.  Return an
  * existing irq if we've already got one for the gsi.
  */
 int xen_allocate_pirq(unsigned gsi, int shareable, char *name)
 {
 	int irq;
-	struct physdev_irq irq_op;
 
 	spin_lock(&irq_mapping_update_lock);
 
@@ -575,20 +563,7 @@ int xen_allocate_pirq(unsigned gsi, int shareable, char *name)
 	set_irq_chip_and_handler_name(irq, &xen_pirq_chip,
 				      handle_level_irq, name);
 
-	irq_op.irq = gsi;
-	irq_op.vector = 0;
-
-	/* Only the privileged domain can do this. For non-priv, the pcifront
-	 * driver provides a PCI bus that does the call to do exactly
-	 * this in the priv domain. */
-	if (xen_initial_domain() &&
-	    HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
-		dynamic_irq_cleanup(irq);
-		irq = -ENOSPC;
- 		goto out;
- 	}
-
-	irq_info[irq] = mk_pirq_info(0, gsi, irq_op.vector);
+	irq_info[irq] = mk_pirq_info(0, gsi);
  	irq_info[irq].u.pirq.flags |= shareable ? PIRQ_SHAREABLE : 0;
 out:
 	spin_unlock(&irq_mapping_update_lock);
@@ -726,7 +701,7 @@ int xen_create_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc,
 		irq = -1;
 		goto out;
 	}
-	irq_info[irq] = mk_pirq_info(0, map_irq.pirq, map_irq.index);
+	irq_info[irq] = mk_pirq_info(0, map_irq.pirq);
 	if (domid)
 		irq_info[irq].u.pirq.domid = domid;
 
@@ -740,11 +715,6 @@ out:
 }
 #endif
 
-int xen_vector_from_irq(unsigned irq)
-{
-	return vector_from_irq(irq);
-}
-
 int xen_gsi_from_irq(unsigned irq)
 {
 	return gsi_from_irq(irq);
diff --git a/include/xen/interface/physdev.h b/include/xen/interface/physdev.h
index ac5de37..39c2b51 100644
--- a/include/xen/interface/physdev.h
+++ b/include/xen/interface/physdev.h
@@ -172,6 +172,16 @@ struct physdev_op {
 	} u;
 };
 
+#define PHYSDEVOP_setup_gsi    21
+struct physdev_setup_gsi {
+    int gsi;
+    /* IN */
+    uint8_t triggering;
+    /* IN */
+    uint8_t polarity;
+    /* IN */
+};
+
 /*
  * Notify that some PIRQ-bound event channels have been unmasked.
  * ** This command is obsolete since interface version 0x00030202 and is **
-- 
1.6.0.rc1


[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/24/09 02:04, Zhang, Xiantao wrote:
>> Shoehorning trig/pol information into it as well is kind of nasty.
>> And I think on any PC system it should suffice to assume GSI 0-15 are
>> ISA edge-triggered active-high, GSI 16+ are PCI level-triggered
>> active-low, and any exceptions are parsed out of MADT or MPBIOS. We
>> pretty much have all that code, it just might need plumbing back in a
>> little bit. Yunhong points out that ACPI DSDT can have overriding
>> objects in the _PRT, but I don't know it ever actually gets used on
>> real-world PC systems. So I would try without, but if we do end up
>> needing to get this info from dom0, I think it should be via a new
>> physdev_op.
>>     
> At least dom0 parses this info from DSDT, so we can't have the assuption whether it is used or not, I think.

Could you clarify this?  Are you saying that Xen can't use the DSDT
because dom0 does?

>  And I also agree to add a new physdev_op to handle this case, and it should be better way to go.  
> Based on this idea, I worked out the patch, attached!  In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each domain can require to map each GSI in this case. 
> In addition, I believe it is very safe to port the hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed.  BTW, I also tested apic and non-apic cases, they works fine after applying the patches. 

Could you resend the Linux patch as a delta against your previous
patch?  It makes it easier both to see how things are evolving, and also
so I can reuse previous merges. 

What branch/version is your diff against?

Do we still need the dummy APIC mappings?  Can anything end up touching
them?

Thanks,
    J

RE: Re: APIC rework

[Zhang, Xiantao]

Jeremy Fitzhardinge wrote:
> On 11/24/09 02:04, Zhang, Xiantao wrote:
>>> Shoehorning trig/pol information into it as well is kind of nasty.
>>> And I think on any PC system it should suffice to assume GSI 0-15
>>> are ISA edge-triggered active-high, GSI 16+ are PCI level-triggered
>>> active-low, and any exceptions are parsed out of MADT or MPBIOS. We
>>> pretty much have all that code, it just might need plumbing back in
>>> a little bit. Yunhong points out that ACPI DSDT can have overriding
>>> objects in the _PRT, but I don't know it ever actually gets used on
>>> real-world PC systems. So I would try without, but if we do end up
>>> needing to get this info from dom0, I think it should be via a new
>>> physdev_op. 
>>> 
>> At least dom0 parses this info from DSDT, so we can't have the
>> assuption whether it is used or not, I think. 
> 
> Could you clarify this?  Are you saying that Xen can't use the DSDT
> because dom0 does?

Dsdt parsing needs AML interpreter, and Xen hasn't this interpreter, so it can't get the info.

>>  And I also agree to add a new physdev_op to handle this case, and
>> it should be better way to go. 
>> Based on this idea, I worked out the patch, attached!  In this
>> patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each
>> GSI setup, and each domain can require to map each GSI in this case.
>> In addition, I believe it is very safe to port the hypervisor patch
>> to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no
>> logic is changed.  BTW, I also tested apic and non-apic cases, they
>> works fine after applying the patches.   
> 
> Could you resend the Linux patch as a delta against your previous
> patch?  It makes it easier both to see how things are evolving, and
> also so I can reuse previous merges.
> 
> What branch/version is your diff against?

I'm against the branch origion/xen/master instead of origin/xen/master-xiantao.  And the base commit is 
"b9160b12ecc71 ". 

> Do we still need the dummy APIC mappings?  Can anything end up
> touching them?

Yes, we still have the dummy mapping in this patch, since ioapic logic and ioapic info(such as GSI info, the info device link with ioapic) parsing are not split separately in current Linux. 
If we want to end up this, we have to modify much logic in upstream. 
Xiantao

Re: Re: APIC rework

[Konrad Rzeszutek Wilk]

> At least dom0 parses this info from DSDT, so we can't have the assuption whether it is used or not, I think. And I also agree to add a new physdev_op to handle this case, and it should be better way to go.  
> Based on this idea, I worked out the patch, attached!  In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each domain can require to map each GSI in this case. 
> In addition, I believe it is very safe to port the hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed.  BTW, I also tested apic and non-apic cases, they works fine after applying the patches. 

But I don't think you tested PCI front and PCI back.

Mainly these lines worry me (can you inline the patch next time too, please):

+               map_irq.domid = DOMID_SELF;                                                                                                
+               map_irq.type = MAP_PIRQ_TYPE_GSI;                                                                                          
+               map_irq.index = gsi;                                                                                                       
+               map_irq.pirq = irq;                                                                                                        
+               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);                                                                  

For PCI passthrough to work, the domid needs to be for the guest domain, while
in this case it is set to Dom0.

There is already a method of extracting the domain id for PCI devices passed to
the guest. Look in the 'xen_create_msi_irq' function.

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/24/09 11:44, Konrad Rzeszutek Wilk wrote:
>> At least dom0 parses this info from DSDT, so we can't have the assuption whether it is used or not, I think. And I also agree to add a new physdev_op to handle this case, and it should be better way to go.  
>> Based on this idea, I worked out the patch, attached!  In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each domain can require to map each GSI in this case. 
>> In addition, I believe it is very safe to port the hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed.  BTW, I also tested apic and non-apic cases, they works fine after applying the patches. 
>>     
> But I don't think you tested PCI front and PCI back.
>
> Mainly these lines worry me (can you inline the patch next time too, please):
>   

(Inline+attach, or an inline attachment rather than plain inline, is
best.  Plain inline with quoted-printable encoding is awkward to deal with.)

> +               map_irq.domid = DOMID_SELF;                                                                                                
> +               map_irq.type = MAP_PIRQ_TYPE_GSI;                                                                                          
> +               map_irq.index = gsi;                                                                                                       
> +               map_irq.pirq = irq;                                                                                                        
> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);                                                                  
>
> For PCI passthrough to work, the domid needs to be for the guest domain, while
> in this case it is set to Dom0.
>
> There is already a method of extracting the domain id for PCI devices passed to
> the guest. Look in the 'xen_create_msi_irq' function.
>   

Hm, I'm not very keen on having xen_create_msi_irq do its own traversal
of xenstore; it should take the domid as a parameter and its caller can
do the walk if necessary.  The direct call makes too much of a direct
dependency between two otherwise unrelated subsystems.

    J

Re: Re: APIC rework

[Konrad Rzeszutek Wilk]

On Tue, Nov 24, 2009 at 03:35:22PM -0800, Jeremy Fitzhardinge wrote:
> On 11/24/09 11:44, Konrad Rzeszutek Wilk wrote:
> >> At least dom0 parses this info from DSDT, so we can't have the assuption whether it is used or not, I think. And I also agree to add a new physdev_op to handle this case, and it should be better way to go.  
> >> Based on this idea, I worked out the patch, attached!  In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each domain can require to map each GSI in this case. 
> >> In addition, I believe it is very safe to port the hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed.  BTW, I also tested apic and non-apic cases, they works fine after applying the patches. 
> >>     
> > But I don't think you tested PCI front and PCI back.
> >
> > Mainly these lines worry me (can you inline the patch next time too, please):
> >   
> 
> (Inline+attach, or an inline attachment rather than plain inline, is
> best.  Plain inline with quoted-printable encoding is awkward to deal with.)
> 
> > +               map_irq.domid = DOMID_SELF;                                                                                                
> > +               map_irq.type = MAP_PIRQ_TYPE_GSI;                                                                                          
> > +               map_irq.index = gsi;                                                                                                       
> > +               map_irq.pirq = irq;                                                                                                        
> > +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);                                                                  
> >
> > For PCI passthrough to work, the domid needs to be for the guest domain, while
> > in this case it is set to Dom0.
> >
> > There is already a method of extracting the domain id for PCI devices passed to
> > the guest. Look in the 'xen_create_msi_irq' function.
> >   
> 
> Hm, I'm not very keen on having xen_create_msi_irq do its own traversal
> of xenstore; it should take the domid as a parameter and its caller can
> do the walk if necessary.  The direct call makes too much of a direct

It would be the easiest way. Unfortunatly the call sequence is the
following:

pciback_do_op
 pciback_enable_msi (dev)
  ... arch_setup_msi_irqs
  ...   arch_setup_msi_irq
  ...     xen_setup_msi_irqs

The caller of the pciback_enable_msi knows of the domain id, but the rest
of the stack does not.  One way of doing this is to provide hooks in event.c, say:

 [de|]register_msi_owner 

(this is akin to how it was done in 2.6.18.hg)

where the device is associated with a specific domain id. We would
call those register/deregister when a new device has changed ownership
to a domain.

Back to the call trace. When we reach xen_setup_msi_irqs we can find the relation.
Lookup the device (on this 'registration list') and see if it is assigned to a
specific domain.

But in essence it is similar in spirit to the mechanism of the xenbus_walk
- that is to find out what domain id is associated with this device.

With the patches posted by Xiantao this behavior would have to be
emulated for PCI INTx devices as well, or some flavour of it.

So the question is, Jeremy, would you prefer to have a '[de|]register_device_owner'
call that both MSI and INTx can use and some way of traversing the list this
creates and finding out if there is an owner for the device? Or just leave
it with the xenbus_walk interface?

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/25/09 06:10, Konrad Rzeszutek Wilk wrote:
> On Tue, Nov 24, 2009 at 03:35:22PM -0800, Jeremy Fitzhardinge wrote:
>   
>> On 11/24/09 11:44, Konrad Rzeszutek Wilk wrote:
>>     
>>>> At least dom0 parses this info from DSDT, so we can't have the assuption whether it is used or not, I think. And I also agree to add a new physdev_op to handle this case, and it should be better way to go.  
>>>> Based on this idea, I worked out the patch, attached!  In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each domain can require to map each GSI in this case. 
>>>> In addition, I believe it is very safe to port the hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed.  BTW, I also tested apic and non-apic cases, they works fine after applying the patches. 
>>>>     
>>>>         
>>> But I don't think you tested PCI front and PCI back.
>>>
>>> Mainly these lines worry me (can you inline the patch next time too, please):
>>>   
>>>       
>> (Inline+attach, or an inline attachment rather than plain inline, is
>> best.  Plain inline with quoted-printable encoding is awkward to deal with.)
>>
>>     
>>> +               map_irq.domid = DOMID_SELF;                                                                                                
>>> +               map_irq.type = MAP_PIRQ_TYPE_GSI;                                                                                          
>>> +               map_irq.index = gsi;                                                                                                       
>>> +               map_irq.pirq = irq;                                                                                                        
>>> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);                                                                  
>>>
>>> For PCI passthrough to work, the domid needs to be for the guest domain, while
>>> in this case it is set to Dom0.
>>>
>>> There is already a method of extracting the domain id for PCI devices passed to
>>> the guest. Look in the 'xen_create_msi_irq' function.
>>>   
>>>       
>> Hm, I'm not very keen on having xen_create_msi_irq do its own traversal
>> of xenstore; it should take the domid as a parameter and its caller can
>> do the walk if necessary.  The direct call makes too much of a direct
>>     
> It would be the easiest way. Unfortunatly the call sequence is the
> following:
>
> pciback_do_op
>  pciback_enable_msi (dev)
>   ... arch_setup_msi_irqs
>   ...   arch_setup_msi_irq
>   ...     xen_setup_msi_irqs
>
> The caller of the pciback_enable_msi knows of the domain id, but the rest
> of the stack does not.  One way of doing this is to provide hooks in event.c, say:
>
>  [de|]register_msi_owner 
>
> (this is akin to how it was done in 2.6.18.hg)
>
> where the device is associated with a specific domain id. We would
> call those register/deregister when a new device has changed ownership
> to a domain.
>
> Back to the call trace. When we reach xen_setup_msi_irqs we can find the relation.
> Lookup the device (on this 'registration list') and see if it is assigned to a
> specific domain.
>
> But in essence it is similar in spirit to the mechanism of the xenbus_walk
> - that is to find out what domain id is associated with this device.
>
> With the patches posted by Xiantao this behavior would have to be
> emulated for PCI INTx devices as well, or some flavour of it.
>
> So the question is, Jeremy, would you prefer to have a '[de|]register_device_owner'
> call that both MSI and INTx can use and some way of traversing the list this
> creates and finding out if there is an owner for the device? Or just leave
> it with the xenbus_walk interface?
>   

I don't particularly object to xenbus_walk per-se, its just that I don't
think xen_create_msi_irq() should call it directly.  It would be OK for
xen_setup_msi_irqs() to call it and pass the results into
xen_create_msi_irq().

Would a registration list be any cleaner?  Presumably you'd just keep a
list of devices being controlled by other domains, so non-presence on
the list means DOMID_SELF, so you'd only need to worry about
registration on pciback paths.  If that could be done without needing to
touch common code (or do the horrible hacks that some of the earlier MSI
patches did), then maybe its worthwhile.

xenbus_walk() ends up actually talking to xenstored, so its fairly
expensive, and adds an opportunity for things to go wrong (it is
effectively doing a call out to dom0 usermode from the depths of the pci
code, which doesn't sound like a terribly healthy activity, but I guess
it has worked OK so far).

    J

Re: Re: APIC rework

[Konrad Rzeszutek Wilk]

> 
> I don't particularly object to xenbus_walk per-se, its just that I don't
> think xen_create_msi_irq() should call it directly.  It would be OK for
> xen_setup_msi_irqs() to call it and pass the results into
> xen_create_msi_irq().
> 
> Would a registration list be any cleaner?  Presumably you'd just keep a
> list of devices being controlled by other domains, so non-presence on
> the list means DOMID_SELF, so you'd only need to worry about
> registration on pciback paths.  If that could be done without needing to
> touch common code (or do the horrible hacks that some of the earlier MSI
> patches did), then maybe its worthwhile.

I think that would work nicely. Should have a patch cooked up shortly
after I am done with the xen-fbfront drier in DomU.

RE: Re: APIC rework

[Zhang, Xiantao]

Konrad Rzeszutek Wilk wrote:
>> At least dom0 parses this info from DSDT, so we can't have the
>> assuption whether it is used or not, I think. And I also agree to
>> add a new physdev_op to handle this case, and it should be better
>> way to go.   
>> Based on this idea, I worked out the patch, attached!  In this
>> patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each
>> GSI setup, and each domain can require to map each GSI in this case.
>> In addition, I believe it is very safe to port the hypervisor patch
>> to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no
>> logic is changed.  BTW, I also tested apic and non-apic cases, they
>> works fine after applying the patches.   
> 
> But I don't think you tested PCI front and PCI back.
> 
> Mainly these lines worry me (can you inline the patch next time too,
> please): 
> 
> +               map_irq.domid = DOMID_SELF;
> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
> +               map_irq.index = gsi;
> +               map_irq.pirq = irq;
> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
> &map_irq); 
> 
> For PCI passthrough to work, the domid needs to be for the guest
> domain, while in this case it is set to Dom0.
> There is already a method of extracting the domain id for PCI devices
> passed to the guest. Look in the 'xen_create_msi_irq' function.

Could you detail the concern ?  This hypercall is only related to GSI, not MSI, why it has side-effect about pci passthrough ? 

Xiantao

Re: Re: APIC rework

[Konrad Rzeszutek Wilk]

On Wed, Nov 25, 2009 at 10:43:47AM +0800, Zhang, Xiantao wrote:
> Konrad Rzeszutek Wilk wrote:
> >> At least dom0 parses this info from DSDT, so we can't have the
> >> assuption whether it is used or not, I think. And I also agree to
> >> add a new physdev_op to handle this case, and it should be better
> >> way to go.   
> >> Based on this idea, I worked out the patch, attached!  In this
> >> patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each
> >> GSI setup, and each domain can require to map each GSI in this case.
> >> In addition, I believe it is very safe to port the hypervisor patch
> >> to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no
> >> logic is changed.  BTW, I also tested apic and non-apic cases, they
> >> works fine after applying the patches.   
> > 
> > But I don't think you tested PCI front and PCI back.
> > 
> > Mainly these lines worry me (can you inline the patch next time too,
> > please): 
> > 
> > +               map_irq.domid = DOMID_SELF;
> > +               map_irq.type = MAP_PIRQ_TYPE_GSI;
> > +               map_irq.index = gsi;
> > +               map_irq.pirq = irq;
> > +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
> > &map_irq); 
> > 
> > For PCI passthrough to work, the domid needs to be for the guest
> > domain, while in this case it is set to Dom0.
> > There is already a method of extracting the domain id for PCI devices
> > passed to the guest. Look in the 'xen_create_msi_irq' function.
> 
> Could you detail the concern ?  This hypercall is only related to GSI, not MSI, why it has side-effect about pci passthrough ? 

This is for PV guests _without_ using QEMU. They are using the PCI backend to "enable"
a device (drivers/xen/pciback and drivers/pci/xen-pcifront.c).
The front and back-end communicate the IRQ number (GSI) to the guest
when enabling a INTx PCI device (not MSI ones).

Then the PV guest can bind the IRQ (GSI) number to its own event channel and
have fully working PCI device.

With your change, the privileged domain pins the device to itself, not to
other domains.

RE: Re: APIC rework

[Zhang, Xiantao]

Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 25, 2009 at 10:43:47AM +0800, Zhang, Xiantao wrote:
>> Konrad Rzeszutek Wilk wrote:
>>>> At least dom0 parses this info from DSDT, so we can't have the
>>>> assuption whether it is used or not, I think. And I also agree to
>>>> add a new physdev_op to handle this case, and it should be better
>>>> way to go. Based on this idea, I worked out the patch, attached! 
>>>> In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi
>>>> for each GSI setup, and each domain can require to map each GSI in
>>>> this case. In addition, I believe it is very safe to port the
>>>> hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running
>>>> on it, since no logic is changed.  BTW, I also tested apic and
>>>> non-apic cases, they works fine after applying the patches.
>>> 
>>> But I don't think you tested PCI front and PCI back.
>>> 
>>> Mainly these lines worry me (can you inline the patch next time
>>> too, please): 
>>> 
>>> +               map_irq.domid = DOMID_SELF;
>>> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
>>> +               map_irq.index = gsi;
>>> +               map_irq.pirq = irq;
>>> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
>>> &map_irq); 
>>> 
>>> For PCI passthrough to work, the domid needs to be for the guest
>>> domain, while in this case it is set to Dom0.
>>> There is already a method of extracting the domain id for PCI
>>> devices passed to the guest. Look in the 'xen_create_msi_irq'
>>> function. 
>> 
>> Could you detail the concern ?  This hypercall is only related to
>> GSI, not MSI, why it has side-effect about pci passthrough ? 
> 
> This is for PV guests _without_ using QEMU. They are using the PCI
> backend to "enable" a device (drivers/xen/pciback and
> drivers/pci/xen-pcifront.c). 
> The front and back-end communicate the IRQ number (GSI) to the guest
> when enabling a INTx PCI device (not MSI ones).
> 
> Then the PV guest can bind the IRQ (GSI) number to its own event
> channel and 
> have fully working PCI device.
> 
> With your change, the privileged domain pins the device to itself,
> not to 
> other domains.

But I think dom0 should own the device first during boot, and then assign it to PV guest when this device is required by pcifront?  Basically, we don't know which devices should be reserved for non-previleged domains, right ? So I think the GSI should be initialized and bind to dom0 when dom0 boots?  Once the devices is assigned to PV guests, it maybe need to do the unmapping operation about the GSI from dom0 and do mapping for the domU. 

BTW, I just met a strange issue with the  function xen_create_msi_irq you mentioned, and it blocks initialization of SR-IOV devices' VFs , and I think it should be a bug.
 
In the function xen_create_msi_irq, there is one line as following to get the domid of the specified device, but a strange domid(0xfff0) is returned by this call, could you help to check whether this strange domid is from?   
domid = rc = xenbus_walk( "/local/domain/0", get_domid_for_dev, dev);
^^^^^^^
Xiantao 

Re: Re: APIC rework

[Konrad Rzeszutek Wilk]

On Wed, Nov 25, 2009 at 11:21:45PM +0800, Zhang, Xiantao wrote:
> Konrad Rzeszutek Wilk wrote:
> > On Wed, Nov 25, 2009 at 10:43:47AM +0800, Zhang, Xiantao wrote:
> >> Konrad Rzeszutek Wilk wrote:
> >>>> At least dom0 parses this info from DSDT, so we can't have the
> >>>> assuption whether it is used or not, I think. And I also agree to
> >>>> add a new physdev_op to handle this case, and it should be better
> >>>> way to go. Based on this idea, I worked out the patch, attached! 
> >>>> In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi
> >>>> for each GSI setup, and each domain can require to map each GSI in
> >>>> this case. In addition, I believe it is very safe to port the
> >>>> hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running
> >>>> on it, since no logic is changed.  BTW, I also tested apic and
> >>>> non-apic cases, they works fine after applying the patches.
> >>> 
> >>> But I don't think you tested PCI front and PCI back.
> >>> 
> >>> Mainly these lines worry me (can you inline the patch next time
> >>> too, please): 
> >>> 
> >>> +               map_irq.domid = DOMID_SELF;
> >>> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
> >>> +               map_irq.index = gsi;
> >>> +               map_irq.pirq = irq;
> >>> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
> >>> &map_irq); 
> >>> 
> >>> For PCI passthrough to work, the domid needs to be for the guest
> >>> domain, while in this case it is set to Dom0.
> >>> There is already a method of extracting the domain id for PCI
> >>> devices passed to the guest. Look in the 'xen_create_msi_irq'
> >>> function. 
> >> 
> >> Could you detail the concern ?  This hypercall is only related to
> >> GSI, not MSI, why it has side-effect about pci passthrough ? 
> > 
> > This is for PV guests _without_ using QEMU. They are using the PCI
> > backend to "enable" a device (drivers/xen/pciback and
> > drivers/pci/xen-pcifront.c). 
> > The front and back-end communicate the IRQ number (GSI) to the guest
> > when enabling a INTx PCI device (not MSI ones).
> > 
> > Then the PV guest can bind the IRQ (GSI) number to its own event
> > channel and 
> > have fully working PCI device.
> > 
> > With your change, the privileged domain pins the device to itself,
> > not to 
> > other domains.
> 
> But I think dom0 should own the device first during boot, and then assign it to PV guest when this device is required by pcifront?  Basically, we don't know which devices should be reserved for non-previleged domains, right ? So I think the GSI should be initialized and bind to dom0 when dom0 boots?  Once the devices is assigned to PV guests, it maybe need to do the unmapping operation about the GSI from dom0 and do mapping for the domU. 

During boot the device can be owned by pciback or by the modele for which a
PCI entry exist. Look for pciback.hide entry.

There are two modes of execution:
 1). First being what you described wherein the device initially belogs to Dom0.
     The user unbinds it from the PCI device and binds it to the pciback module.
     At that point, the device is disabled and ready for PV guests. When a PV guest
     starts pciback module makes the pci_enable_device call and sets the IRQ, etc
     for the device (for MSI, it obviously gets the IDT value from the hypervisor
     call).
 2). Dom0 boots where the user specified on the command line pciback.hide=(0000:01:03.1).
     The pciback owns the device (is binded to it) and the native module that would
     load for this PCI device is not called.
    
It is correct that the unmapping/mapping and the ownership needs to be dynamic. As user
could bind to the pciback module, give it to guest, kill the guest, then map the PCI device
back to dom0, and after that repeat the whole thing.
 
> 
> BTW, I just met a strange issue with the  function xen_create_msi_irq you mentioned, and it blocks initialization of SR-IOV devices' VFs , and I think it should be a bug.

Hmm.. I am not sure if this is the appropiate place for it. You see
this driver is designed for the machines that don't have SR-IOV, VFs,
or VT-d to be able to passthrough a PCI device to the PV guest. You can
use this to run PV guests on Pentium 4 style machine.

I think that the SR-IOV devices would go through a different call stack
to enable the device? Either way, I recently got my hands on a SR-IOV machine
and will see how this works.

Has the 2.6.31.5 been working before with SR-IOV cards or is this
your first experiment with this.

>  
> In the function xen_create_msi_irq, there is one line as following to get the domid of the specified device, but a strange domid(0xfff0) is returned by this call, could you help to check whether this strange domid is from?   
> domid = rc = xenbus_walk( "/local/domain/0", get_domid_for_dev, dev);

That is due to casting (domid is short int, rc is int).
You need to check if rc is negative, and if so set the domid to DOMID_SELF.
The next line below does this:

 if (domid <= 0)
                domid = DOMID_SELF;

Ohh wait, looks like the fix for this never got pushed upstream. That
should have been 'if (rc <= 0)'. Yikes.

rc is -16 (EBUSY). That implies that xenstored has not
been initialized on Dom0.


> ^^^^^^^
> Xiantao 
> 

RE: Re: APIC rework

[Zhang, Xiantao]

Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 25, 2009 at 11:21:45PM +0800, Zhang, Xiantao wrote:
>> Konrad Rzeszutek Wilk wrote:
>>> On Wed, Nov 25, 2009 at 10:43:47AM +0800, Zhang, Xiantao wrote:
>>>> Konrad Rzeszutek Wilk wrote:
>>>>>> At least dom0 parses this info from DSDT, so we can't have the
>>>>>> assuption whether it is used or not, I think. And I also agree to
>>>>>> add a new physdev_op to handle this case, and it should be better
>>>>>> way to go. Based on this idea, I worked out the patch, attached!
>>>>>> In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi
>>>>>> for each GSI setup, and each domain can require to map each GSI
>>>>>> in this case. In addition, I believe it is very safe to port the
>>>>>> hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running
>>>>>> on it, since no logic is changed.  BTW, I also tested apic and
>>>>>> non-apic cases, they works fine after applying the patches.
>>>>> 
>>>>> But I don't think you tested PCI front and PCI back.
>>>>> 
>>>>> Mainly these lines worry me (can you inline the patch next time
>>>>> too, please): 
>>>>> 
>>>>> +               map_irq.domid = DOMID_SELF;
>>>>> +               map_irq.type = MAP_PIRQ_TYPE_GSI;
>>>>> +               map_irq.index = gsi;
>>>>> +               map_irq.pirq = irq;
>>>>> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq,
>>>>> &map_irq); 
>>>>> 
>>>>> For PCI passthrough to work, the domid needs to be for the guest
>>>>> domain, while in this case it is set to Dom0.
>>>>> There is already a method of extracting the domain id for PCI
>>>>> devices passed to the guest. Look in the 'xen_create_msi_irq'
>>>>> function.
>>>> 
>>>> Could you detail the concern ?  This hypercall is only related to
>>>> GSI, not MSI, why it has side-effect about pci passthrough ?
>>> 
>>> This is for PV guests _without_ using QEMU. They are using the PCI
>>> backend to "enable" a device (drivers/xen/pciback and
>>> drivers/pci/xen-pcifront.c). The front and back-end communicate the
>>> IRQ number (GSI) to the guest when enabling a INTx PCI device (not
>>> MSI ones). 
>>> 
>>> Then the PV guest can bind the IRQ (GSI) number to its own event
>>> channel and have fully working PCI device.
>>> 
>>> With your change, the privileged domain pins the device to itself,
>>> not to other domains.
>> 
>> But I think dom0 should own the device first during boot, and then
>> assign it to PV guest when this device is required by pcifront? 
>> Basically, we don't know which devices should be reserved for
>> non-previleged domains, right ? So I think the GSI should be
>> initialized and bind to dom0 when dom0 boots?  Once the devices is
>> assigned to PV guests, it maybe need to do the unmapping operation
>> about the GSI from dom0 and do mapping for the domU.      
> 
> During boot the device can be owned by pciback or by the modele for
> which a 
> PCI entry exist. Look for pciback.hide entry.
> 
> There are two modes of execution:
>  1). First being what you described wherein the device initially
>      belogs to Dom0. The user unbinds it from the PCI device and
>      binds it to the pciback module. At that point, the device is
>      disabled and ready for PV guests. When a PV guest starts pciback
>      module makes the pci_enable_device call and sets the IRQ, etc
>      for the device (for MSI, it obviously gets the IDT value from
>  the hypervisor call). 2). Dom0 boots where the user specified on the
>      command line pciback.hide=(0000:01:03.1). The pciback owns the
>      device (is binded to it) and the native module that would load
> for this PCI device is not called. 
> 
> It is correct that the unmapping/mapping and the ownership needs to
> be dynamic. As user could bind to the pciback module, give it to
> guest, kill the guest, then map the PCI device back to dom0, and
> after that repeat the whole thing.

If only the above two cases are needed to support, I think my patch should meet the requirement. This is because once the device is hidden by the pciback, the pirq unmap logic should be callled in both cases. So it the mapping for dom0 should be cleared in the operation, so there is not the issue you figured.  :-)
Xiantao 

Re: Re: APIC rework

[Konrad Rzeszutek Wilk]

> > During boot the device can be owned by pciback or by the modele for
> > which a 
> > PCI entry exist. Look for pciback.hide entry.
> > 
> > There are two modes of execution:
> >  1). First being what you described wherein the device initially
> >      belogs to Dom0. The user unbinds it from the PCI device and
> >      binds it to the pciback module. At that point, the device is
> >      disabled and ready for PV guests. When a PV guest starts pciback
> >      module makes the pci_enable_device call and sets the IRQ, etc
> >      for the device (for MSI, it obviously gets the IDT value from
> >  the hypervisor call). 2). Dom0 boots where the user specified on the
> >      command line pciback.hide=(0000:01:03.1). The pciback owns the
> >      device (is binded to it) and the native module that would load
> > for this PCI device is not called. 
> > 
> > It is correct that the unmapping/mapping and the ownership needs to
> > be dynamic. As user could bind to the pciback module, give it to
> > guest, kill the guest, then map the PCI device back to dom0, and
> > after that repeat the whole thing.
> 
> If only the above two cases are needed to support, I think my patch should meet the requirement. This is because once the device is hidden by the pciback, the pirq unmap logic should be callled in both cases. So it the mapping for dom0 should be cleared in the operation, so there is not the issue you figured.  :-)

Ah, what about maping the device to a guest? That is not neccessary? Or are you
saying that the PV guest should be making a hypervisor call to PHYSDEVOP_map_pirq and
it will map the device itself? I thought those calls in the Hypervisor
were guarded by a IS_PRIV(xx) which would expel a PV guest. The other complication is
that the PV guests have their own PCI subsystem (look in arch/x86/pci/xen.c)
which only calls 'xen_allocate_pirq' to setup the chip_data to xen_pirq_chip.

The next call the PV guest does is to startup_pirq.

> Xiantao 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: Re: APIC rework

[Zhang, Xiantao]

Konrad Rzeszutek Wilk wrote:
>>> During boot the device can be owned by pciback or by the modele for
>>> which a PCI entry exist. Look for pciback.hide entry.
>>> 
>>> There are two modes of execution:
>>>  1). First being what you described wherein the device initially
>>>      belogs to Dom0. The user unbinds it from the PCI device and
>>>      binds it to the pciback module. At that point, the device is
>>>      disabled and ready for PV guests. When a PV guest starts
>>>      pciback module makes the pci_enable_device call and sets the
>>>      IRQ, etc for the device (for MSI, it obviously gets the IDT
>>>  value from the hypervisor call). 2). Dom0 boots where the user
>>>      specified on the command line pciback.hide=(0000:01:03.1). The
>>>      pciback owns the device (is binded to it) and the native
>>> module that would load 
>>> for this PCI device is not called.
>>> 
>>> It is correct that the unmapping/mapping and the ownership needs to
>>> be dynamic. As user could bind to the pciback module, give it to
>>> guest, kill the guest, then map the PCI device back to dom0, and
>>> after that repeat the whole thing.
>> 
>> If only the above two cases are needed to support, I think my patch
>> should meet the requirement. This is because once the device is
>> hidden by the pciback, the pirq unmap logic should be callled in
>> both cases. So it the mapping for dom0 should be cleared in the
>> operation, so there is not the issue you figured.  :-)    
> 
> Ah, what about maping the device to a guest? That is not neccessary?
> Or are you saying that the PV guest should be making a hypervisor
> call to PHYSDEVOP_map_pirq and it will map the device itself? I
> thought those calls in the Hypervisor 
> were guarded by a IS_PRIV(xx) which would expel a PV guest. The other
> complication is that the PV guests have their own PCI subsystem (look
> in arch/x86/pci/xen.c) 
> which only calls 'xen_allocate_pirq' to setup the chip_data to
> xen_pirq_chip. 
> 
> The next call the PV guest does is to startup_pirq.

I am also confusing about how to assign a device to a PV guest. Originally  I thought the logic should be it should be hidden first through pciback in dom0, and pv guest uses its pci subsystem (pcifront end) to require pciback to mapping this device.  In the first step, once pciback.hide logic is called in dom0,  this device should be released by dom0 and avaliable for PV guests at this time. And the in subsequent step, pciback or dom0's pci system should help PV guests to do the irq mapping, otherwise I can't see how the irqs from the assigned device are delivered to PV guests.   

Xiantao

Re: Re: APIC rework

[Konrad Rzeszutek Wilk]

> I am also confusing about how to assign a device to a PV guest. Originally  I thought the logic should be it should be hidden first through pciback in dom0, and pv guest uses its pci subsystem (pcifront end) to require pciback to mapping this device.  In the first step, once pciback.hide logic is called in dom0,  this device should be released by dom0 and avaliable for PV guests at this time. And the in subsequent step, pciback or dom0's pci system should help PV guests to do the irq mapping, otherwise I can't see how the irqs from the assigned device are delivered to PV guests.   

That is correct. Here is an URL of the correct steps:
http://wiki.xensource.com/xenwiki/Assign_hardware_to_DomU_with_PCIBack_as_module

>From the unprivileged side (domU), when it makes a call to
pci_enable_device, it gets routed to pcifront_bus_write.
On the privileged side (dom0), pciback picks up the write
and routes it to 'command_write' (conf_space_header.c). It
does the pci_enable_device in the dom0 side. The pci_enable_device
ends up calling xen_allocate_pirq which gets the IRQ from
PHYSDEVOP_alloc_irq_vector. That IRQ is saved in dev->irq and
is visible to the DomU.

Also during the pci_enable_device (in the DomU side), pcibios_enable_device
gets called - which in domU is called 'xen_pcifront_enable_irq'.
The xen_pcifront_enable_irq allocates an irq_desc with xen_pirq_chip
structure. The GSI it requests is actually the IRQ number from dev->irq.

To summarize, dom0 on behalf of domU, calls PHYSDEVOP_alloc_irq_vector
for the device in question. Saves the GSI in dev->irq which is visible to
the DomU. DomU sets up a xen_pirq_chip structure for the device and
starts/stop/etc through that function structure. Please note that 
there is nothing in PHYSDEVOP_alloc..  about which domain owns the device.
That is only done with PHYSDEVOP_map_pirq calls.

With your patch instead of PHYSDEVOP_alloc_irq_vector, it would be
PHYSDEVOP_map_pirq.

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/25/09 07:21, Zhang, Xiantao wrote:
> In the function xen_create_msi_irq, there is one line as following to get the domid of the specified device, but a strange domid(0xfff0) is returned by this call, could you help to check whether this strange domid is from?   
> domid = rc = xenbus_walk( "/local/domain/0", get_domid_for_dev, dev);
>   

That's DOMID_SELF.

    J

RE: Re: APIC rework

[Zhang, Xiantao]

Jeremy Fitzhardinge wrote:
> On 11/25/09 07:21, Zhang, Xiantao wrote:
>> In the function xen_create_msi_irq, there is one line as following
>> to get the domid of the specified device, but a strange
>> domid(0xfff0) is returned by this call, could you help to check
>> whether this strange domid is from? domid = rc = xenbus_walk(
>> "/local/domain/0", get_domid_for_dev, dev);   
>> 
> 
> That's DOMID_SELF.

No, DOMID_SELF is 0x7ff0, not 0xfff0, and it should be -EBUSY, and they are accidentally similar. :)
Xiantao 

Re: Re: APIC rework

[Konrad Rzeszutek Wilk]

On Tue, Nov 17, 2009 at 08:46:19PM +0800, Zhang, Xiantao wrote:
> Jeremy Fitzhardinge wrote:
> > On 11/16/09 19:45, Keir Fraser wrote:
> >> It's kind of a shame to need this though. Is there no way for the
> >> hypervisor to work out automatically whether an older dom0 is
> >> running? Or work out the trigger/level stuff for itself (after all
> >> it parses the relevant bios tables just like dom0)?
> > 
> > If Xen can set the interrupt triggering by itself, why would it ever
> > need dom0 to do it?  Couldn't it just preconfigure all the pins, and
> > then wait for dom0 to provide/request the pirq<->evtchn mapping?
> 
> 
> After reviewing the logic, I think we can use DOMID_SELF to identify new dom0.  In linux-2.6.18 dom0, only qemu uses this hypercall for device assginment, so map->domid shouldn't be dom0.  If old dom0/qemu with this hypercall, keeps the logic unchanged, and only change the logic for new dom0 when call into it.   Attached the patch. 

What about privileged domains that are not domain 0? Won't that
mess this up?

RE: Re: APIC rework

[Zhang, Xiantao]

Konrad Rzeszutek Wilk wrote:
> On Tue, Nov 17, 2009 at 08:46:19PM +0800, Zhang, Xiantao wrote:
>> Jeremy Fitzhardinge wrote:
>>> On 11/16/09 19:45, Keir Fraser wrote:
>>>> It's kind of a shame to need this though. Is there no way for the
>>>> hypervisor to work out automatically whether an older dom0 is
>>>> running? Or work out the trigger/level stuff for itself (after all
>>>> it parses the relevant bios tables just like dom0)?
>>> 
>>> If Xen can set the interrupt triggering by itself, why would it ever
>>> need dom0 to do it?  Couldn't it just preconfigure all the pins, and
>>> then wait for dom0 to provide/request the pirq<->evtchn mapping?
>> 
>> 
>> After reviewing the logic, I think we can use DOMID_SELF to identify
>> new dom0.  In linux-2.6.18 dom0, only qemu uses this hypercall for
>> device assginment, so map->domid shouldn't be dom0.  If old
>> dom0/qemu with this hypercall, keeps the logic unchanged, and only
>> change the logic for new dom0 when call into it.   Attached the
>> patch.     
> 
> What about privileged domains that are not domain 0? Won't that
> mess this up?

Do you mean driver domain, I think the driver domain should use DOMID_SELF for the hypercall. :)
Xiantao

Re: Re: APIC rework

[Jeremy Fitzhardinge]

On 11/16/09 19:13, Zhang, Xiantao wrote:
> Jeremy Fitzhardinge wrote:
>> Is there any way for the dom0 kernel to tell whether the hypervisor is
>> implementing the new ABI, so it can choose how to set up interrupts.
>>
>> MAP_COMPAT_BIT doesn't seem like a very good name, because it implies
>> that setting it reverts to "compatible" behaviour.  I assume that
>> leaving it clear enables the historical behaviour and setting it
>> enables the new one (since old kernels won't be setting it).
>>     
> Maybe better change to MAP_NEW_ABI_BIT ?  Since the hypervisor patch didn't change old code path after introducing this bit, so it is very easy and safe to backport to xen-3.4-testing tree, and make new dom0 be able to run top of it.  :)
>   

Better to give it a name which actually describes what it means/does. 
NEW_ABI isn't very descriptive when it becomes *the* ABI (and what
happens when there's a newer one?).

    J
> Xiantao
>   

Re: Announcing xen/master: pvops git trees rearranged

[Christian Tramnitz]

Jeremy Fitzhardinge wrote:
> No plans to put anything into .32.  We need to have a solid story about
> how to handle IOAPIC setup before pushing the rest, I think.  I've just
> restarted work on that, but I need to work out how to reconcile it with
> the recent MSI work.
> 
>     J

Yes, I guess this increases the likelihood of things being merged...

On another note, what about xen-tip/next? Wasn't there pciback (or was 
it pcifront) already incorporated (or was that wishful thinking)? I 
haven't seen any changes there for month, is it not updated anymore?


Best regards,
    Christian

Re: Re: Announcing xen/master: pvops git trees rearranged

[Andy Burns]

2009/9/24 Christian Tramnitz <chris.ace@gmx.net>:

> Jeremy Fitzhardinge wrote:
>>
>> No plans to put anything into .32.  We need to have a solid story about
>> how to handle IOAPIC setup before pushing the rest, I think.
>
> Yes, I guess this increases the likelihood of things being merged...

One potentially disturbing snippet on LWN today ...

http://lwn.net/Articles/353852/

Re: Re: Announcing xen/master: pvops git trees rearranged

[Thiago Camargo Martins Cordeiro]

[-- Attachment #1.1: Type: text/plain, Size: 984 bytes --]

I don't believe that, because the paravirt_ops provides a great
virtualization job without the need for a special CPU. There are too many
people around the world in this situation...I don't have too much CPUs with
VMX and in all my dom0s that have the VMX flag, I simple disable it in my
BIOS.

The mainline Linux with the pv_ops dom0 will arrive some day... for sure!

Cheers,
Thiago

2009/9/24 Andy Burns <xen.lists@burns.me.uk>

> 2009/9/24 Christian Tramnitz <chris.ace@gmx.net>:
>
> > Jeremy Fitzhardinge wrote:
> >>
> >> No plans to put anything into .32.  We need to have a solid story about
> >> how to handle IOAPIC setup before pushing the rest, I think.
> >
> > Yes, I guess this increases the likelihood of things being merged...
>
> One potentially disturbing snippet on LWN today ...
>
> http://lwn.net/Articles/353852/
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 1702 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Announcing xen/master: pvops git trees rearranged

[Thiago Camargo Martins Cordeiro]

[-- Attachment #1.1: Type: text/plain, Size: 1224 bytes --]

And what about the OpenSolaris, NetBSD and FreeBSD paravirtualized kernels?!Who
needs the full virtualization?!

:-D

Thiago

2009/9/24 Thiago Camargo Martins Cordeiro <thiagocmartinsc@gmail.com>

> I don't believe that, because the paravirt_ops provides a great
> virtualization job without the need for a special CPU. There are too many
> people around the world in this situation...I don't have too much CPUs
> with VMX and in all my dom0s that have the VMX flag, I simple disable it in
> my BIOS.
>
> The mainline Linux with the pv_ops dom0 will arrive some day... for sure!
>
> Cheers,
> Thiago
>
> 2009/9/24 Andy Burns <xen.lists@burns.me.uk>
>
> 2009/9/24 Christian Tramnitz <chris.ace@gmx.net>:
>>
>> > Jeremy Fitzhardinge wrote:
>> >>
>> >> No plans to put anything into .32.  We need to have a solid story about
>> >> how to handle IOAPIC setup before pushing the rest, I think.
>> >
>> > Yes, I guess this increases the likelihood of things being merged...
>>
>> One potentially disturbing snippet on LWN today ...
>>
>> http://lwn.net/Articles/353852/
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>
>

[-- Attachment #1.2: Type: text/html, Size: 2259 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Announcing xen/master: pvops git trees rearranged

[Jeremy Fitzhardinge]

On 09/24/09 10:47, Andy Burns wrote:
> One potentially disturbing snippet on LWN today ...
>
> http://lwn.net/Articles/353852/
>   

I wouldn't read too much into that.  At some point in the (distant)
future we may decide that pvops doesn't have enough users to justify
supporting it any more and pull it out, like any other kernel feature
(or perhaps more specifically, nobody's interested in maintaining it any
more).  But we're a long way from there.  After all, people are still
actively maintaining 80386 support...

    J

Re: Re: Announcing xen/master: pvops git trees rearranged

[Jeremy Fitzhardinge]

On 09/24/09 06:20, Christian Tramnitz wrote:
> Yes, I guess this increases the likelihood of things being merged...

Right.

> On another note, what about xen-tip/next? Wasn't there pciback (or was
> it pcifront) already incorporated (or was that wishful thinking)?

There are a lot of pieces needed for pci back/front in the tree already,
but it just needs some work to glue them all together.  I think Konrad
is going to look at it shortly.

> I haven't seen any changes there for month, is it not updated anymore?

No, xen-tip has been dead for a while.   All the action is on xen/master.

    J

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge wrote:
> 
> This is definitely a work-in-progress kernel.  I'd appreciate all bug
> *and* success reports so I can get some idea of how many people are
> using this thing, and how often there are problems.  Patches gratefully
> accepted.
> 

I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on x86_64 AHCI box.

The good news is that the dom0 kernel boots up, but there are some error
messages.

Using the default options (modeset) the VGA console doesn't work, it
goes blank (display says "power save") in the beginning of dom0 kernel boot:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt

When using "nomodeset" dom0 kernel option the VGA text console works:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-nomodeset-2009-10-11.txt

Both of the above logs have really long errors/tracebacks about:
[ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]

The latest Fedora 12/rawhide myoung xendom0 kernel has the same errors:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64.txt

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Konrad Rzeszutek Wilk]

On Sun, Oct 11, 2009 at 06:39:00PM +0300, Pasi Kärkkäinen wrote:
> On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge wrote:
> > 
> > This is definitely a work-in-progress kernel.  I'd appreciate all bug
> > *and* success reports so I can get some idea of how many people are
> > using this thing, and how often there are problems.  Patches gratefully
> > accepted.
> > 
> 
> I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on x86_64 AHCI box.
> 
> The good news is that the dom0 kernel boots up, but there are some error
> messages.
> 
> Using the default options (modeset) the VGA console doesn't work, it
> goes blank (display says "power save") in the beginning of dom0 kernel boot:
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt

This line:
[drm:radeon_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)

Is a pretty good pointer at what the fault is. If you look at git commit
93e7c3850b8431e19c9cba91413066bfd2360671 you will see the band-aid Jeremy added.
It looks though as if not all of the radeon drivers allocate their ring buffer memory via
drm_sg_alloc calls thought. Not sure how the r100 (and the corresponding X driver) does it.
The long/erro traceback about the HARDIRQ is a red-herring in this case.

Here is a couple of things that I would like you to try, if you can:

1). Pass in 'drm.debug=255' and send the output. It should have tons of extra
output. 

2). Send in the Xorg.log (or whatever output the program in the userland that
 starts the modesetting produces).  I don't have much knowledge in how modesetting works,
 so this might require some digging.

2). When you boot the kernel without Xen, you don't see this error, right?

3) What does lspci -vvv output show for the video card in question?

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Mon, Oct 12, 2009 at 04:02:48PM -0400, Konrad Rzeszutek Wilk wrote:
> On Sun, Oct 11, 2009 at 06:39:00PM +0300, Pasi Kärkkäinen wrote:
> > On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge wrote:
> > > 
> > > This is definitely a work-in-progress kernel.  I'd appreciate all bug
> > > *and* success reports so I can get some idea of how many people are
> > > using this thing, and how often there are problems.  Patches gratefully
> > > accepted.
> > > 
> > 
> > I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on x86_64 AHCI box.
> > 
> > The good news is that the dom0 kernel boots up, but there are some error
> > messages.
> > 
> > Using the default options (modeset) the VGA console doesn't work, it
> > goes blank (display says "power save") in the beginning of dom0 kernel boot:
> > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt
> 
> This line:
> [drm:radeon_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)
> 
> Is a pretty good pointer at what the fault is. If you look at git commit
> 93e7c3850b8431e19c9cba91413066bfd2360671 you will see the band-aid Jeremy added.
> It looks though as if not all of the radeon drivers allocate their ring buffer memory via
> drm_sg_alloc calls thought. Not sure how the r100 (and the corresponding X driver) does it.
> The long/erro traceback about the HARDIRQ is a red-herring in this case.
> 
> Here is a couple of things that I would like you to try, if you can:
> 

Sure.

> 1). Pass in 'drm.debug=255' and send the output. It should have tons of extra
> output. 
> 

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-14-drmdebug.txt

Unknown boot option `drm.debug=255': ignoring

Also it seems this error:
[ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]

.. comes from the USB stuff.

I tried Fedora xendom0 kernel rpm, and the kernel graphics modesetting
seems to work there! (Fedora kernel contains newer graphics/drm drivers).

But the same USB related error is there with the fedora kernel:
[ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt


> 2). Send in the Xorg.log (or whatever output the program in the userland that
>  starts the modesetting produces).  I don't have much knowledge in how modesetting works,
>  so this might require some digging.
> 

Hmm.. yeah, I'm not sure either which is the first program setting up
graphics mode using kernel modesetting (KMS) in Fedora.. 

I extracted the initrd image and checked the 'init' script:

echo "Loading drm module"
modprobe -q drm
echo "Loading ttm module"
modprobe -q ttm
echo "Loading radeon module"
modprobe -q radeon
/lib/udev/console_init tty0
plymouth --show-splash

So I guess plymouth is asking for a graphics mode..


> 2). When you boot the kernel without Xen, you don't see this error, right?
> 

Yeah, it works OK on baremetal without Xen.
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-baremetal.txt

> 3) What does lspci -vvv output show for the video card in question?
> 

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/lspci-vvv-2.6.31.1-baremetal.txt
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/lspci-vvv-2.6.31.1-dom0.txt

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Konrad Rzeszutek Wilk]

On Thu, Oct 15, 2009 at 12:14:15AM +0300, Pasi Kärkkäinen wrote:
> On Mon, Oct 12, 2009 at 04:02:48PM -0400, Konrad Rzeszutek Wilk wrote:
> > On Sun, Oct 11, 2009 at 06:39:00PM +0300, Pasi Kärkkäinen wrote:
> > > On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge wrote:
> > > > 
> > > > This is definitely a work-in-progress kernel.  I'd appreciate all bug
> > > > *and* success reports so I can get some idea of how many people are
> > > > using this thing, and how often there are problems.  Patches gratefully
> > > > accepted.
> > > > 
> > > 
> > > I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on x86_64 AHCI box.
> > > 
> > > The good news is that the dom0 kernel boots up, but there are some error
> > > messages.
> > > 
> > > Using the default options (modeset) the VGA console doesn't work, it
> > > goes blank (display says "power save") in the beginning of dom0 kernel boot:
> > > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt
> > 
> > This line:
> > [drm:radeon_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)
> > 
> > Is a pretty good pointer at what the fault is. If you look at git commit
> > 93e7c3850b8431e19c9cba91413066bfd2360671 you will see the band-aid Jeremy added.
> > It looks though as if not all of the radeon drivers allocate their ring buffer memory via
> > drm_sg_alloc calls thought. Not sure how the r100 (and the corresponding X driver) does it.
> > The long/erro traceback about the HARDIRQ is a red-herring in this case.
> > 
> > Here is a couple of things that I would like you to try, if you can:
> > 
> 
> Sure.
> 
> > 1). Pass in 'drm.debug=255' and send the output. It should have tons of extra
> > output. 
> > 
> 
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-14-drmdebug.txt
> 
> Unknown boot option `drm.debug=255': ignoring

I forgot to mention that you probably need to have CONFIG_DRM set to 'y' instead of 'm'
for this to work. Or you could hack up the initrd (modprobe.conf) and make drm load
with the 'debug=255' parameter.

.. snip ..

> seems to work there! (Fedora kernel contains newer graphics/drm drivers).
> 
> But the same USB related error is there with the fedora kernel:
> [ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]
> 
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt

Nah. Still has the same problem:

[drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)
[drm:r100_cp_init] *ERROR* radeon: cp isn't working (-

> 
> 
> > 2). Send in the Xorg.log (or whatever output the program in the userland that
> >  starts the modesetting produces).  I don't have much knowledge in how modesetting works,
> >  so this might require some digging.
> > 
> 
> Hmm.. yeah, I'm not sure either which is the first program setting up
> graphics mode using kernel modesetting (KMS) in Fedora.. 
> 
> I extracted the initrd image and checked the 'init' script:
> 
> echo "Loading drm module"
> modprobe -q drm
> echo "Loading ttm module"
> modprobe -q ttm
> echo "Loading radeon module"
> modprobe -q radeon
> /lib/udev/console_init tty0

add here:
export LIBGL_DEBUG=verbose

> plymouth --show-splash
> 
> So I guess plymouth is asking for a graphics mode..

Add this to your kernel command line: plymouth:debug

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Thu, Oct 15, 2009 at 04:04:09PM -0400, Konrad Rzeszutek Wilk wrote:
> > > 
> > > Here is a couple of things that I would like you to try, if you can:
> > > 
> > 
> > Sure.
> > 
> > > 1). Pass in 'drm.debug=255' and send the output. It should have tons of extra
> > > output. 
> > > 
> > 
> > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-14-drmdebug.txt
> > 
> > Unknown boot option `drm.debug=255': ignoring
> 
> I forgot to mention that you probably need to have CONFIG_DRM set to 'y' instead of 'm'
> for this to work. Or you could hack up the initrd (modprobe.conf) and make drm load
> with the 'debug=255' parameter.
>

It seems there are two separate issues; the radeon/drm failure, and then the USB problem.
More about USB later in this mail.

I hacked the initrd and added 'debug=255' when modprobing drm.ko.
Log below.

> 
> > seems to work there! (Fedora kernel contains newer graphics/drm drivers).
> > 
> > But the same USB related error is there with the fedora kernel:
> > [ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]
> > 
> > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt
> 
> Nah. Still has the same problem:
> 
> [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)
> [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-
> 

True, I missed that. But the modesetting actually works on that fedora kernel, so it's
probably driver version related, since fedora has newer gfx stuff than
upstream kernel (some of the drm/radeon developers are fedora developers so the patches 
end up in fedora kernels before upstream kernels).

> > 
> > 
> > > 2). Send in the Xorg.log (or whatever output the program in the userland that
> > >  starts the modesetting produces).  I don't have much knowledge in how modesetting works,
> > >  so this might require some digging.
> > > 
> > 
> > Hmm.. yeah, I'm not sure either which is the first program setting up
> > graphics mode using kernel modesetting (KMS) in Fedora.. 
> > 
> > I extracted the initrd image and checked the 'init' script:
> > 
> > echo "Loading drm module"
> > modprobe -q drm
> > echo "Loading ttm module"
> > modprobe -q ttm
> > echo "Loading radeon module"
> > modprobe -q radeon
> > /lib/udev/console_init tty0
> 
> add here:
> export LIBGL_DEBUG=verbose
> 

Done.

> > plymouth --show-splash
> > 
> > So I guess plymouth is asking for a graphics mode..
> 
> Add this to your kernel command line: plymouth:debug

Done.

Log with the modifications:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-16-drmdebug.txt

Still same problems related to radeon:
[drm:radeon_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)
[drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22).


Then more about USB stuff:

usb 1-5: new high speed USB device using ehci_hcd and address 2
<huge list of backtraces>

I think those USB devices are the IPMI management devices for remote KVM
(keyboard/video/mouse).. shouldn't make any difference I guess..


-- Pasi

[drm:r100_ring_test] *ERROR* radeon: ring test failed

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 2391 bytes --]

> > > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt
> > 
> > Nah. Still has the same problem:
> > 
> > [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)
> > [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-

I looked a bit at the code and tried to reproduce this, but found out that the hardware I've is
a bit too modern for mode-setting to work (no support yet for RS780).

I looked at the code yesterday and I think I've found the
the failure. But I don't have yet a test bed for this, so if you
are willing to be guinea pig, please test this patch (also attached):


diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index b8b6c4a..4445364 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -27,7 +27,7 @@
 /*
  * Authors: Thomas Hellstrom <thellstrom-at-vmware-dot-com>
  */
-
+#include <linux/dma-mapping.h>
 #include <linux/vmalloc.h>
 #include <linux/sched.h>
 #include <linux/highmem.h>
@@ -138,6 +138,9 @@ static void ttm_tt_free_page_directory(struct ttm_tt *ttm)
 static struct page *ttm_tt_alloc_page(unsigned page_flags)
 {
 	gfp_t gfp_flags = GFP_USER;
+	void *addr;
+	dma_addr_t phys;
+	struct page *page;
 
 	if (page_flags & TTM_PAGE_FLAG_ZERO_ALLOC)
 		gfp_flags |= __GFP_ZERO;
@@ -147,7 +150,24 @@ static struct page *ttm_tt_alloc_page(unsigned page_flags)
 	else
 		gfp_flags |= __GFP_HIGHMEM;
 
-	return alloc_page(gfp_flags);
+	addr = dma_alloc_coherent(NULL, PAGE_SIZE, &phys, gfp_flags);
+	BUG_ON(!addr);
+	page = virt_to_page(addr);
+	get_page(page);
+	return page;
+}
+
+static void ttm_tt_free_page(struct page *page)
+{
+	void *addr;
+
+ 	if (page == NULL)
+		return;
+
+	put_page(page);
+	addr = page_address(page);
+
+	dma_free_coherent(NULL, PAGE_SIZE, addr, virt_to_bus(addr));
 }
 
 static void ttm_tt_free_user_pages(struct ttm_tt *ttm)
@@ -180,7 +200,7 @@ static void ttm_tt_free_user_pages(struct ttm_tt *ttm)
 
 		ttm->pages[i] = NULL;
 		ttm_mem_global_free(ttm->bdev->mem_glob, PAGE_SIZE, false);
-		put_page(page);
+		ttm_tt_free_page(page);
 	}
 	ttm->state = tt_unpopulated;
 	ttm->first_himem_page = ttm->num_pages;
@@ -218,7 +238,7 @@ static struct page *__ttm_tt_get_page(struct ttm_tt *ttm, int index)
 	}
 	return p;
 out_err:
-	put_page(p);
+	ttm_tt_free_page(p);
 	return NULL;
 }
 

[-- Attachment #2: ttm.patch --]
[-- Type: text/plain, Size: 2136 bytes --]

diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index b8b6c4a..4445364 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -27,7 +27,7 @@
 /*
  * Authors: Thomas Hellstrom <thellstrom-at-vmware-dot-com>
  */
-
+#include <linux/dma-mapping.h>
 #include <linux/vmalloc.h>
 #include <linux/sched.h>
 #include <linux/highmem.h>
@@ -138,6 +138,9 @@ static void ttm_tt_free_page_directory(struct ttm_tt *ttm)
 static struct page *ttm_tt_alloc_page(unsigned page_flags)
 {
 	gfp_t gfp_flags = GFP_USER;
+	void *addr;
+	dma_addr_t phys;
+	struct page *page;
 
 	if (page_flags & TTM_PAGE_FLAG_ZERO_ALLOC)
 		gfp_flags |= __GFP_ZERO;
@@ -147,7 +150,24 @@ static struct page *ttm_tt_alloc_page(unsigned page_flags)
 	else
 		gfp_flags |= __GFP_HIGHMEM;
 
-	return alloc_page(gfp_flags);
+	addr = dma_alloc_coherent(NULL, PAGE_SIZE, &phys, gfp_flags);
+	BUG_ON(!addr);
+	page = virt_to_page(addr);
+	get_page(page);
+	return page;
+}
+
+static void ttm_tt_free_page(struct page *page)
+{
+	void *addr;
+
+ 	if (page == NULL)
+		return;
+
+	put_page(page);
+	addr = page_address(page);
+
+	dma_free_coherent(NULL, PAGE_SIZE, addr, virt_to_bus(addr));
 }
 
 static void ttm_tt_free_user_pages(struct ttm_tt *ttm)
@@ -180,7 +200,7 @@ static void ttm_tt_free_user_pages(struct ttm_tt *ttm)
 
 		ttm->pages[i] = NULL;
 		ttm_mem_global_free(ttm->bdev->mem_glob, PAGE_SIZE, false);
-		put_page(page);
+		ttm_tt_free_page(page);
 	}
 	ttm->state = tt_unpopulated;
 	ttm->first_himem_page = ttm->num_pages;
@@ -218,7 +238,7 @@ static struct page *__ttm_tt_get_page(struct ttm_tt *ttm, int index)
 	}
 	return p;
 out_err:
-	put_page(p);
+	ttm_tt_free_page(p);
 	return NULL;
 }
 
diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h
index a68829d..e57e442 100644
--- a/include/drm/ttm/ttm_bo_driver.h
+++ b/include/drm/ttm/ttm_bo_driver.h
@@ -524,7 +524,7 @@ extern void ttm_tt_destroy(struct ttm_tt *ttm);
 extern void ttm_tt_unbind(struct ttm_tt *ttm);
 
 /**
- * ttm_ttm_destroy:
+ * ttm_tt_get_page:
  *
  * @ttm: The struct ttm_tt.
  * @index: Index of the desired page.

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Tue, Oct 20, 2009 at 12:58:05PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt
> > > 
> > > Nah. Still has the same problem:
> > > 
> > > [drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)
> > > [drm:r100_cp_init] *ERROR* radeon: cp isn't working (-
> 
> I looked a bit at the code and tried to reproduce this, but found out that the hardware I've is
> a bit too modern for mode-setting to work (no support yet for RS780).
> 
> I looked at the code yesterday and I think I've found the
> the failure. But I don't have yet a test bed for this, so if you
> are willing to be guinea pig, please test this patch (also attached):
> 

I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
today, and also applied your ttm.patch.

Modesetting works now, and there are no drm/radeon errors.
USB related errors/tracebacks remain though.

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.4-2009-10-21-with-ttm-patch.txt

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Konrad Rzeszutek Wilk]

> I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
> today, and also applied your ttm.patch.
> 
> Modesetting works now, and there are no drm/radeon errors.

Thank you for testing it.

> USB related errors/tracebacks remain though.

You saw this on bare-metal too, right?

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk wrote:
> > I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
> > today, and also applied your ttm.patch.
> > 
> > Modesetting works now, and there are no drm/radeon errors.
> 
> Thank you for testing it.
> 

No problems. Thanks for the patch :)

> > USB related errors/tracebacks remain though.
> 
> You saw this on bare-metal too, right?
>

Nope. The same kernel booted on baremetal (without Xen) works OK without
the USB errors/tracebacks:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.4-2009-10-21-baremetal.txt

I only get the USB tracebacks on pv_ops dom0 kernel.

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Jeremy Fitzhardinge]

>
> Nope. The same kernel booted on baremetal (without Xen) works OK without
> the USB errors/tracebacks:
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.4-2009-10-21-baremetal.txt
>
> I only get the USB tracebacks on pv_ops dom0 kernel.
>   

Yeah, that's a known bug at the moment.  It should be harmless.

(I need to fix up vm_unmap_aliases() to be callable in interrupt context.)

    J

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Wed, Oct 21, 2009 at 12:50:08PM -0700, Jeremy Fitzhardinge wrote:
> >
> > Nope. The same kernel booted on baremetal (without Xen) works OK without
> > the USB errors/tracebacks:
> > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.4-2009-10-21-baremetal.txt
> >
> > I only get the USB tracebacks on pv_ops dom0 kernel.
> >   
> 
> Yeah, that's a known bug at the moment.  It should be harmless.
> 
> (I need to fix up vm_unmap_aliases() to be callable in interrupt context.)
> 

Ok, thanks for the info.

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk wrote:
> > I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
> > today, and also applied your ttm.patch.
> > 
> > Modesetting works now, and there are no drm/radeon errors.
> 
> Thank you for testing it.
> 

Btw are you going to post this for inclusion in drm/ttm trees?

Or should it be committed to pvops tree?

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]

On Tue, Oct 27, 2009 at 05:46:51PM +0200, Pasi Kärkkäinen wrote:
> On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk wrote:
> > > I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
> > > today, and also applied your ttm.patch.
> > > 
> > > Modesetting works now, and there are no drm/radeon errors.
> > 
> > Thank you for testing it.
> > 
> 
> Btw are you going to post this for inclusion in drm/ttm trees?

I am not really comfortable with it. It has the same drawbacks
as the fix for the drm_scatter, where we blindly assume
phys_to_bus(virt_to_phys(X)) will give us the same value as
what dma_alloc_coherent provides. We should save that bus address
somewhere...

Saving it somewhere (perhaps in some of the structs the drm_ttm allocates)
could do it. But we should probably differentiate between memory
that is being allocated for DMA transfers vs other things so that
we don't over-exercise the dma_alloc_coherent. Thought maybe
the memory returned via drm_tt calls are only used for DMA transfers.

We can figure this out.  Pasi, I don't have a modesetting working machine,
but you do. Can you compile your pv_ops with the fix I provided earlier,
along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is turned
on and your machine is humming along (maybe even run glxgears), compile
the attached module and load it. You should get a kernel dump
off all devices that are using the DMA buffers. Can you e-mail me that back
please?



[-- Attachment #2: Makefile --]
[-- Type: text/plain, Size: 758 bytes --]

# Comment/uncomment the following line to disable/enable debugging
#DEBUG = y

# Add your debugging flag (or not) to CFLAGS
ifeq ($(DEBUG),y)
  DEBFLAGS = -O -g # "-O" is needed to expand inlines
else
  DEBFLAGS = -O2
endif

EXTRA_CFLAGS += $(DEBFLAGS) -I$(LDDINCDIR)

ifneq ($(KERNELRELEASE),)
# call from kernel build system

obj-m	:= dump_dma.o

else

#KERNELDIR ?= /lib/modules/$(shell uname -r)/build
KERNELDIR ?= /home/konrad/git/nex-VI-PVOPS/linux-build
PWD       := $(shell pwd)

default:
	$(MAKE) -C $(KERNELDIR) M=$(PWD) LDDINCDIR=$(PWD)/../include modules

endif

clean:
	rm -rf *.o *~ core .depend .*.cmd *.ko *.mod.c .tmp_versions

depend .depend dep:
	$(CC) $(CFLAGS) -M *.c > .depend


ifeq (.depend,$(wildcard .depend))
include .depend
endif

[-- Attachment #3: dump_dma.c --]
[-- Type: text/plain, Size: 1851 bytes --]

/*
 *  Copyright 2009 Virtual Iron.
 *  by Konrad Rzeszutek <konrad@virtualiron.com>
 *
 * This code exposes the buffer-cache to userland via sysfs.
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License v2.0 as published by
 * the Free Software Foundation
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 */

#include <linux/module.h>
#include <linux/string.h>
#include <linux/types.h>
#include <linux/init.h>
#include <linux/stat.h>
#include <linux/err.h>
#include <linux/ctype.h>
#include <linux/slab.h>
#include <linux/limits.h>
#include <linux/device.h>
#include <linux/pci.h>
#include <linux/blkdev.h>
#include <linux/device.h>

#include <linux/init.h>
#include <linux/mm.h>
#include <linux/fcntl.h>
#include <linux/slab.h>
#include <linux/kmod.h>
#include <linux/major.h>
#include <linux/smp_lock.h>
#include <linux/highmem.h>
#include <linux/blkdev.h>
#include <linux/module.h>
#include <linux/blkpg.h>
#include <linux/buffer_head.h>
#include <linux/mpage.h>
#include <linux/mount.h>
#include <linux/uio.h>
#include <linux/namei.h>
#include <asm/uaccess.h>

#include <linux/pagemap.h>
#include <linux/pagevec.h>
#include <linux/buffer_head.h>

#include <linux/dma-debug.h>
#define DUMP_DMA_VERSION  "0.1"

MODULE_AUTHOR("Konrad Rzeszutek <konrad@virtualiron>");
MODULE_DESCRIPTION("dumps DMA stats");
MODULE_LICENSE("GPL");
MODULE_VERSION(DUMP_DMA_VERSION);

static int __init dma_dump_init(void)
{
	dump_stack();
	debug_dma_dump_mappings(NULL);
	return -ENODEV;
}

static void __exit dma_dump_exit(void)
{
}

module_init(dma_dump_init);
module_exit(dma_dump_exit);

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Tue, Oct 27, 2009 at 01:00:19PM -0400, Konrad Rzeszutek Wilk wrote:
> On Tue, Oct 27, 2009 at 05:46:51PM +0200, Pasi Kärkkäinen wrote:
> > On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
> > > > today, and also applied your ttm.patch.
> > > > 
> > > > Modesetting works now, and there are no drm/radeon errors.
> > > 
> > > Thank you for testing it.
> > > 
> > 
> > Btw are you going to post this for inclusion in drm/ttm trees?
> 
> I am not really comfortable with it. It has the same drawbacks
> as the fix for the drm_scatter, where we blindly assume
> phys_to_bus(virt_to_phys(X)) will give us the same value as
> what dma_alloc_coherent provides. We should save that bus address
> somewhere...
> 
> Saving it somewhere (perhaps in some of the structs the drm_ttm allocates)
> could do it. But we should probably differentiate between memory
> that is being allocated for DMA transfers vs other things so that
> we don't over-exercise the dma_alloc_coherent. Thought maybe
> the memory returned via drm_tt calls are only used for DMA transfers.
>

Ok..

> We can figure this out.  Pasi, I don't have a modesetting working machine,
> but you do. Can you compile your pv_ops with the fix I provided earlier,
> along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is turned
> on and your machine is humming along (maybe even run glxgears), compile
> the attached module and load it. You should get a kernel dump
> off all devices that are using the DMA buffers. Can you e-mail me that back
> please?
> 

Yeah, I'll do that and get back to you.

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Tue, Oct 27, 2009 at 01:00:19PM -0400, Konrad Rzeszutek Wilk wrote:
> On Tue, Oct 27, 2009 at 05:46:51PM +0200, Pasi Kärkkäinen wrote:
> > On Wed, Oct 21, 2009 at 02:31:30PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > I updated the pv_ops dom0 git tree to the latest 2.6.31.4 tree as of
> > > > today, and also applied your ttm.patch.
> > > > 
> > > > Modesetting works now, and there are no drm/radeon errors.
> > > 
> > > Thank you for testing it.
> > > 
> > 
> > Btw are you going to post this for inclusion in drm/ttm trees?
> 
> I am not really comfortable with it. It has the same drawbacks
> as the fix for the drm_scatter, where we blindly assume
> phys_to_bus(virt_to_phys(X)) will give us the same value as
> what dma_alloc_coherent provides. We should save that bus address
> somewhere...
> 
> Saving it somewhere (perhaps in some of the structs the drm_ttm allocates)
> could do it. But we should probably differentiate between memory
> that is being allocated for DMA transfers vs other things so that
> we don't over-exercise the dma_alloc_coherent. Thought maybe
> the memory returned via drm_tt calls are only used for DMA transfers.
> 
> We can figure this out.  Pasi, I don't have a modesetting working machine,
> but you do. Can you compile your pv_ops with the fix I provided earlier,
> along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is turned
> on and your machine is humming along (maybe even run glxgears), compile
> the attached module and load it. You should get a kernel dump
> off all devices that are using the DMA buffers. Can you e-mail me that back
> please?
> 
> 

I've never actually tried X on this box.. it seems X doesn't start.

http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.4-2009-10-27.txt
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/Xorg.0.log-2009-10-27

X never comes up.. screen goes blank/black, but there still is some signal on the monitor.. 
and I can't switch VTs from the console keyboard.

"kill <pid_of_X>" doesn't help.. it seems only reboot helps.

Hmm.. I wonder how to debug this. I guess X is stuck trying to determine
modelines from the (non-connected) DVI display.. actually I don't even have
DVI connector on the motherboard..

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Konrad Rzeszutek Wilk]

> > We can figure this out.  Pasi, I don't have a modesetting working machine,
> > but you do. Can you compile your pv_ops with the fix I provided earlier,
> > along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is turned
> > on and your machine is humming along (maybe even run glxgears), compile
> > the attached module and load it. You should get a kernel dump
> > off all devices that are using the DMA buffers. Can you e-mail me that back
> > please?
> > 
> > 
> 
> I've never actually tried X on this box.. it seems X doesn't start.

As in, even on baremetal it does not work?
> 
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.4-2009-10-27.txt
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/Xorg.0.log-2009-10-27

This is weird. The EDID reporting seems to be stuck in a endless loop:

(II) RADEON(0): EDID for output DVI-0
(II) RADEON(0): EDID for output VGA-0
...
(II) RADEON(0): EDID for output DVI-0
(II) RADEON(0): EDID for output VGA-0
..
and then last one where it is cut off (probably when you rebooted the box?)

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Tue, Oct 27, 2009 at 03:41:14PM -0400, Konrad Rzeszutek Wilk wrote:
> > > We can figure this out.  Pasi, I don't have a modesetting working machine,
> > > but you do. Can you compile your pv_ops with the fix I provided earlier,
> > > along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is turned
> > > on and your machine is humming along (maybe even run glxgears), compile
> > > the attached module and load it. You should get a kernel dump
> > > off all devices that are using the DMA buffers. Can you e-mail me that back
> > > please?
> > > 
> > > 
> > 
> > I've never actually tried X on this box.. it seems X doesn't start.
> 
> As in, even on baremetal it does not work?

Heh. should have tried that already. Too tired atm :)

Anyway, I tried baremetal now. X comes up, I can see the desktop, and
then the monitor goes out of signal.

"dmesg" is full of:

[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
[TTM] Erroneous page count. Leaking pages.
X used greatest stack depth: 3000 bytes left

Could that be caused by the patch? I booted the same ttm-patched kernel
on baremetal without Xen..

> > 
> > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.4-2009-10-27.txt
> > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/Xorg.0.log-2009-10-27
> 
> This is weird. The EDID reporting seems to be stuck in a endless loop:
> 
> (II) RADEON(0): EDID for output DVI-0
> (II) RADEON(0): EDID for output VGA-0
> ...
> (II) RADEON(0): EDID for output DVI-0
> (II) RADEON(0): EDID for output VGA-0
> ..
> and then last one where it is cut off (probably when you rebooted the box?)


Xorg.0.log for baremetal looks pretty much the same:
http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/Xorg.0.log-2009-10-27-baremetal

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Konrad Rzeszutek Wilk]

> > As in, even on baremetal it does not work?
> 
> Heh. should have tried that already. Too tired atm :)
> 
> Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> then the monitor goes out of signal.
> 
> "dmesg" is full of:
> 
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> X used greatest stack depth: 3000 bytes left
> 
> Could that be caused by the patch? I booted the same ttm-patched kernel

I fear so. 

> on baremetal without Xen..

By next week I should have a working machine where I can do the mode-setting
and dig some deeper in this.

Thanks so far for testing some ideas out.

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Tue, Oct 27, 2009 at 04:13:06PM -0400, Konrad Rzeszutek Wilk wrote:
> > > As in, even on baremetal it does not work?
> > 
> > Heh. should have tried that already. Too tired atm :)
> > 
> > Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> > then the monitor goes out of signal.
> > 
> > "dmesg" is full of:
> > 
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > X used greatest stack depth: 3000 bytes left
> > 
> > Could that be caused by the patch? I booted the same ttm-patched kernel
> 
> I fear so. 
> 
> > on baremetal without Xen..
> 
> By next week I should have a working machine where I can do the mode-setting
> and dig some deeper in this.
> 
> Thanks so far for testing some ideas out.
>

Ok. Just let me know if you need me to test something else/more.

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Tue, Oct 27, 2009 at 04:13:06PM -0400, Konrad Rzeszutek Wilk wrote:
> > > As in, even on baremetal it does not work?
> > 
> > Heh. should have tried that already. Too tired atm :)
> > 
> > Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> > then the monitor goes out of signal.
> > 
> > "dmesg" is full of:
> > 
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > [TTM] Erroneous page count. Leaking pages.
> > X used greatest stack depth: 3000 bytes left
> > 
> > Could that be caused by the patch? I booted the same ttm-patched kernel
> 
> I fear so. 
> 
> > on baremetal without Xen..
> 
> By next week I should have a working machine where I can do the mode-setting
> and dig some deeper in this.
> 
> Thanks so far for testing some ideas out.


Hello again and happy new year!

Konrad: Have you had time to look at this yet? 

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Konrad Rzeszutek Wilk]

On Fri, Jan 01, 2010 at 07:21:37PM +0200, Pasi Kärkkäinen wrote:
> On Tue, Oct 27, 2009 at 04:13:06PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > As in, even on baremetal it does not work?
> > > 
> > > Heh. should have tried that already. Too tired atm :)
> > > 
> > > Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> > > then the monitor goes out of signal.
> > > 
> > > "dmesg" is full of:
> > > 
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > [TTM] Erroneous page count. Leaking pages.
> > > X used greatest stack depth: 3000 bytes left
> > > 
> > > Could that be caused by the patch? I booted the same ttm-patched kernel
> > 
> > I fear so. 
> > 
> > > on baremetal without Xen..
> > 
> > By next week I should have a working machine where I can do the mode-setting
> > and dig some deeper in this.
> > 
> > Thanks so far for testing some ideas out.
> 
> 
> Hello again and happy new year!

Heya!
> 
> Konrad: Have you had time to look at this yet? 

Uhh, ... umm.. Thank you for reminding me to look at it. I am right now
doing a experimental swiotlb that I want to post to iommu-devel and see what
they think about it.

After that I am going back to revisit this bug.

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Mon, Jan 04, 2010 at 08:37:28AM -0500, Konrad Rzeszutek Wilk wrote:
> On Fri, Jan 01, 2010 at 07:21:37PM +0200, Pasi Kärkkäinen wrote:
> > On Tue, Oct 27, 2009 at 04:13:06PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > > As in, even on baremetal it does not work?
> > > > 
> > > > Heh. should have tried that already. Too tired atm :)
> > > > 
> > > > Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> > > > then the monitor goes out of signal.
> > > > 
> > > > "dmesg" is full of:
> > > > 
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > [TTM] Erroneous page count. Leaking pages.
> > > > X used greatest stack depth: 3000 bytes left
> > > > 
> > > > Could that be caused by the patch? I booted the same ttm-patched kernel
> > > 
> > > I fear so. 
> > > 
> > > > on baremetal without Xen..
> > > 
> > > By next week I should have a working machine where I can do the mode-setting
> > > and dig some deeper in this.
> > > 
> > > Thanks so far for testing some ideas out.
> > 
> > 
> > Hello again and happy new year!
> 
> Heya!
> > 
> > Konrad: Have you had time to look at this yet? 
> 
> Uhh, ... umm.. Thank you for reminding me to look at it. I am right now
> doing a experimental swiotlb that I want to post to iommu-devel and see what
> they think about it.
> 
> After that I am going back to revisit this bug.
>

Ok, thanks for the update :)
Let me know if/when you have some patch to test.

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Konrad Rzeszutek Wilk]

> Ok, thanks for the update :)
> Let me know if/when you have some patch to test.

Trying to refresh my memory. The issue was that under Xen, the DRM
modesetting with your radeon card (ES1000) would fail during the
ring buffer initialization. I provided a test-path that made it go
away, but now you get those Free Memory leak.

So is the Memory leak happening after running X for some time or
pretty much immediately?

Thanks.

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Thu, Jan 14, 2010 at 03:05:48PM -0500, Konrad Rzeszutek Wilk wrote:
> > Ok, thanks for the update :)
> > Let me know if/when you have some patch to test.
> 
> Trying to refresh my memory. The issue was that under Xen, the DRM
> modesetting with your radeon card (ES1000) would fail during the
> ring buffer initialization. I provided a test-path that made it go
> away, but now you get those Free Memory leak.
> 

Let's see.

You sent me this patch:
http://lists.xensource.com/archives/html/xen-devel/2009-10/msg00986.html

And then kernel modesetting started working:
http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01014.html

Then you asked me to try running X and getting some debug stuff, but it
seems X still doesn't work for me:
http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01309.html

Some analysis and it seems the patch causes TTM "Erroneous page count. Leaking pages." errors:
http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01316.html

> So is the Memory leak happening after running X for some time or
> pretty much immediately?
> 

Starting X causes the errors.

If I don't start X then the system runs fine without errors. Setting the
graphics mode during boot/initrd works OK.

I can re-rest things if needed.

-- Pasi

Re: Re: [drm:r100_ring_test] *ERROR* radeon: ring test failed

[Pasi Kärkkäinen]

On Tue, Oct 27, 2009 at 10:18:32PM +0200, Pasi Kärkkäinen wrote:
> On Tue, Oct 27, 2009 at 03:41:14PM -0400, Konrad Rzeszutek Wilk wrote:
> > > > We can figure this out.  Pasi, I don't have a modesetting working machine,
> > > > but you do. Can you compile your pv_ops with the fix I provided earlier,
> > > > along with enabling CONFIG_DMA_API_DEBUG=y. Once mode-setting is turned
> > > > on and your machine is humming along (maybe even run glxgears), compile
> > > > the attached module and load it. You should get a kernel dump
> > > > off all devices that are using the DMA buffers. Can you e-mail me that back
> > > > please?
> > > > 
> > > > 
> > > 
> > > I've never actually tried X on this box.. it seems X doesn't start.
> > 
> > As in, even on baremetal it does not work?
> 
> Heh. should have tried that already. Too tired atm :)
> 
> Anyway, I tried baremetal now. X comes up, I can see the desktop, and
> then the monitor goes out of signal.
> 
> "dmesg" is full of:
> 
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> [TTM] Erroneous page count. Leaking pages.
> X used greatest stack depth: 3000 bytes left
> 
> Could that be caused by the patch? I booted the same ttm-patched kernel
> on baremetal without Xen..
> 

Using the 2.6.31.5-96.fc12.x86_64 fedora kernel rpm (baremetal) 
X works OK. No TTM errors in dmesg.

Xorg.0.log looks similar.. last line is:

(II) RADEON(0): EDID for output DVI-0

So I guess the Xorg.0.log part is how it should be.. 

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Stefan Kuhne]

[-- Attachment #1.1: Type: text/plain, Size: 398 bytes --]

Jeremy Fitzhardinge schrieb:

Hello,

> Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
> was a fairly nasty bug which ended up corrupting dom0 memory when doing
> IO on behalf of domains, but that is finally fixed.
> 
runs this Kernel with xen-3.3.1 or does it need xen.-3.4.x?
My xen-3.3.1 tells me that my Kernel ist noc ELF binary.

Regards,
Stefan Kuhne


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 552 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Announcing xen/master: pvops git trees rearranged

[Pasi Kärkkäinen]

On Fri, Dec 04, 2009 at 05:07:32PM +0100, Stefan Kuhne wrote:
> Jeremy Fitzhardinge schrieb:
> 
> Hello,
> 
> > Well, my current pvops/dom0 tree is finally (reasonably) stable.  There
> > was a fairly nasty bug which ended up corrupting dom0 memory when doing
> > IO on behalf of domains, but that is finally fixed.
> > 
> runs this Kernel with xen-3.3.1 or does it need xen.-3.4.x?
> My xen-3.3.1 tells me that my Kernel ist noc ELF binary.
> 

Please read this wiki page:
http://wiki.xensource.com/xenwiki/XenParavirtOps

Especially the "Xen requirements for using pv_ops dom0 kernel" chapter.

-- Pasi

Re: Announcing xen/master: pvops git trees rearranged

[Jeremy Fitzhardinge]

On 12/04/09 08:07, Stefan Kuhne wrote:
> runs this Kernel with xen-3.3.1 or does it need xen.-3.4.x?
> My xen-3.3.1 tells me that my Kernel ist noc ELF binary.
>    

If you're using an old Xen, you need to use the vmlinux(.gz) file as 
your kernel image.  Newer versions of Xen can directly load the bzImage.

     J

Re: Announcing xen/master: pvops git trees rearranged

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 8540 bytes --]


Dmesg report for 2.6.31.4 been built on F12 and loaded under Xen 3.4.1 (installed via rawhide 3.4.1-5 ) Dom0 on top of F12 ( yum updated)

[drm] Initialized drm 1.1.0 20060810
[drm] radeon default to kernel modesetting.
[drm] radeon kernel modesetting enabled.
xen: registering gsi 16 triggering 0 polarity 1
xen_allocate_pirq: returning irq 16 for gsi 16
xen: --> irq=16
xen_set_ioapic_routing: irq 16 gsi 16 vector 152 ioapic 0 pin 16 triggering 1 polarity 1
radeon 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
radeon 0000:01:00.0: setting latency timer to 64
[drm] radeon: Initializing kernel modesetting.
[drm:radeon_driver_load_kms] *ERROR* Failed to initialize radeon, disabling IOCTL
radeon 0000:01:00.0: PCI INT A disabled
radeon: probe of 0000:01:00.0 failed with error -22

.   .   .   .   .   .


======================================================
[ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]
2.6.31.4 #2
------------------------------------------------------
khubd/28 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
 (&retval->lock){......}, at: [<ffffffff81126fa4>] dma_pool_alloc+0x46/0x312

and this task is already holding:
 (&ehci->lock){-.....}, at: [<ffffffff814359e4>] ehci_urb_enqueue+0xb4/0xd7c
which would create a new lock dependency:
 (&ehci->lock){-.....} -> (&retval->lock){......}

but this new dependency connects a HARDIRQ-irq-safe lock:
 (&ehci->lock){-.....}
... which became HARDIRQ-irq-safe at:
  [<ffffffff810996d8>] __lock_acquire+0x256/0xc11
  [<ffffffff8109a181>] lock_acquire+0xee/0x12e
  [<ffffffff81579e9f>] _spin_lock+0x45/0x8e
  [<ffffffff814345ec>] ehci_irq+0x41/0x441
  [<ffffffff814195d5>] usb_hcd_irq+0x59/0xcc
  [<ffffffff810c8200>] handle_IRQ_event+0x62/0x148
  [<ffffffff810ca797>] handle_level_irq+0x90/0xf9
  [<ffffffff81018038>] handle_irq+0x9a/0xba
  [<ffffffff81302342>] xen_evtchn_do_upcall+0x10c/0x1bd
  [<ffffffff8101623e>] xen_do_hypervisor_callback+0x1e/0x30
  [<ffffffffffffffff>] 0xffffffffffffffff

to a HARDIRQ-irq-unsafe lock:
 (purge_lock){+.+...}
... which became HARDIRQ-irq-unsafe at:
...  [<ffffffff8109974d>] __lock_acquire+0x2cb/0xc11
  [<ffffffff8109a181>] lock_acquire+0xee/0x12e
  [<ffffffff81579e9f>] _spin_lock+0x45/0x8e
  [<ffffffff81120137>] __purge_vmap_area_lazy+0x63/0x198
  [<ffffffff81121a15>] vm_unmap_aliases+0x18f/0x1b2
  [<ffffffff8100e400>] xen_alloc_ptpage+0x47/0x75
  [<ffffffff8100e46b>] xen_alloc_pte+0x13/0x15
  [<ffffffff81115495>] __pte_alloc_kernel+0x6f/0xdd
  [<ffffffff81120f42>] vmap_page_range_noflush+0x1c5/0x315
  [<ffffffff811210d3>] map_vm_area+0x41/0x6b
  [<ffffffff8112122c>] __vmalloc_area_node+0x12f/0x167
  [<ffffffff811212f4>] __vmalloc_node+0x90/0xb5
  [<ffffffff81121169>] __vmalloc_area_node+0x6c/0x167
  [<ffffffff811212f4>] __vmalloc_node+0x90/0xb5
  [<ffffffff8112156b>] __vmalloc+0x28/0x3e
  [<ffffffff81adb40a>] alloc_large_system_hash+0x12f/0x1fb
  [<ffffffff81addc9a>] vfs_caches_init+0xb8/0x140
  [<ffffffff81ab5a69>] start_kernel+0x3ef/0x44c
  [<ffffffff81ab4d70>] x86_64_start_reservations+0xbb/0xd6
  [<ffffffff81ab93b7>] xen_start_kernel+0x5d5/0x5dc
  [<ffffffffffffffff>] 0xffffffffffffffff

other info that might help us debug this:

2 locks held by khubd/28:
 #0:  (usb_address0_mutex){+.+...}, at: [<ffffffff81414344>] hub_port_init+0x8c/0x81e
 #1:  (&ehci->lock){-.....}, at: [<ffffffff814359e4>] ehci_urb_enqueue+0xb4/0xd7c

the HARDIRQ-irq-safe lock's dependencies:
-> (&ehci->lock){-.....} ops: 0 {
   IN-HARDIRQ-W at:
                        [<ffffffff810996d8>] __lock_acquire+0x256/0xc11
                        [<ffffffff8109a181>] lock_acquire+0xee/0x12e
                        [<ffffffff81579e9f>] _spin_lock+0x45/0x8e
                        [<ffffffff814345ec>] ehci_irq+0x41/0x441
                        [<ffffffff814195d5>] usb_hcd_irq+0x59/0xcc
                        [<ffffffff810c8200>] handle_IRQ_event+0x62/0x148
                        [<ffffffff810ca797>] handle_level_irq+0x90/0xf9
                        [<ffffffff81018038>] handle_irq+0x9a/0xba
                        [<ffffffff81302342>] xen_evtchn_do_upcall+0x10c/0x1bd
                        [<ffffffff8101623e>] xen_do_hypervisor_callback+0x1e/0x30
                        [<ffffffffffffffff>] 0xffffffffffffffff
.  .  .  .

The most recent build 2.6.31.1 on F12 produced clean dmesg output.
 Builds 2.6.31.4 ( same commit on top) on F11 and Ubuntu 9.04 Server seem
clean.

Boris.

--- On Thu, 10/15/09, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Subject: Re: [Xen-devel] Announcing xen/master: pvops git trees rearranged
To: "Pasi Kärkkäinen" <pasik@iki.fi>
Cc: "Jeremy Fitzhardinge" <jeremy@goop.org>, "Xen-devel" <xen-devel@lists.xensource.com>
Date: Thursday, October 15, 2009, 4:04 PM

On Thu, Oct 15, 2009 at 12:14:15AM +0300, Pasi Kärkkäinen wrote:
> On Mon, Oct 12, 2009 at 04:02:48PM -0400, Konrad Rzeszutek Wilk wrote:
> > On Sun, Oct 11, 2009 at 06:39:00PM +0300, Pasi Kärkkäinen wrote:
> > > On Fri, Sep 18, 2009 at 06:19:41PM -0700, Jeremy Fitzhardinge wrote:
> > > > 
> > > > This is definitely a work-in-progress kernel.  I'd appreciate all bug
> > > > *and* success reports so I can get some idea of how many people are
> > > > using this thing, and how often there are problems.  Patches gratefully
> > > > accepted.
> > > > 
> > > 
> > > I just tried the latest pv_ops dom0 git tree (11 Oct 2009) on x86_64 AHCI box.
> > > 
> > > The good news is that the dom0 kernel boots up, but there are some error
> > > messages.
> > > 
> > > Using the default options (modeset) the VGA console doesn't work, it
> > > goes blank (display says "power save") in the beginning of dom0 kernel boot:
> > > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/dmesg-2.6.31.1-2009-10-11.txt
> > 
> > This line:
> > [drm:radeon_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)
> > 
> > Is a pretty good pointer at what the fault is. If you look at git commit
> > 93e7c3850b8431e19c9cba91413066bfd2360671 you will see the band-aid Jeremy added.
> > It looks though as if not all of the radeon drivers allocate their ring buffer memory via
> > drm_sg_alloc calls thought. Not sure how the r100 (and the corresponding X driver) does it.
> > The long/erro traceback about the HARDIRQ is a red-herring in this case.
> > 
> > Here is a couple of things that I would like you to try, if you can:
> > 
> 
> Sure.
> 
> > 1). Pass in 'drm.debug=255' and send the output. It should have tons of extra
> > output. 
> > 
> 
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.1-2009-10-14-drmdebug.txt
> 
> Unknown boot option `drm.debug=255': ignoring

I forgot to mention that you probably need to have CONFIG_DRM set to 'y' instead of 'm'
for this to work. Or you could hack up the initrd (modprobe.conf) and make drm load
with the 'debug=255' parameter.

.. snip ..

> seems to work there! (Fedora kernel contains newer graphics/drm drivers).
> 
> But the same USB related error is there with the fedora kernel:
> [ INFO: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected ]
> 
> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/radeondebug/dmesg-2.6.31.3-1.2.71.xendom0.fc12.x86_64-2009-10-14.txt

Nah. Still has the same problem:

[drm:r100_ring_test] *ERROR* radeon: ring test failed (sracth(0x15E4)=0xCAFEDEAD)
[drm:r100_cp_init] *ERROR* radeon: cp isn't working (-

> 
> 
> > 2). Send in the Xorg.log (or whatever output the program in the userland that
> >  starts the modesetting produces).  I don't have much knowledge in how modesetting works,
> >  so this might require some digging.
> > 
> 
> Hmm.. yeah, I'm not sure either which is the first program setting up
> graphics mode using kernel modesetting (KMS) in Fedora.. 
> 
> I extracted the initrd image and checked the 'init' script:
> 
> echo "Loading drm module"
> modprobe -q drm
> echo "Loading ttm module"
> modprobe -q ttm
> echo "Loading radeon module"
> modprobe -q radeon
> /lib/udev/console_init tty0

add here:
export LIBGL_DEBUG=verbose

> plymouth --show-splash
> 
> So I guess plymouth is asking for a graphics mode..

Add this to your kernel command line: plymouth:debug

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 12128 bytes --]

[-- Attachment #2: dmesg.2afad356210ac35cbbc81904e0ac8b514b7f6212.gz --]
[-- Type: application/x-gzip, Size: 16663 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel