upstream merge status for 2.6.35, .36?

upstream merge status for 2.6.35, .36?

[Josip Rodin]

Hi,

.35-rc1 has been tagged in mainline for a while now, and I noticed now that
two particular bug fixes from Ian's repo were added, so it looks like .35
will be bugfix only, too. If so, can someone update the wiki please?

What about the future? I saw Konrad's applied his swiotlb tree with the
right acks for inclusion into linux-next, so that looks like it's planned
to be ready to go in when the .36 merge window opens, right?

-- 
     2. That which causes joy or happiness.

Re: upstream merge status for 2.6.35, .36?

[Jeremy Fitzhardinge]

On 06/04/2010 03:39 PM, Josip Rodin wrote:
> What about the future? I saw Konrad's applied his swiotlb tree with the
> right acks for inclusion into linux-next, so that looks like it's planned
> to be ready to go in when the .36 merge window opens, right?
>   

Yes, and I'm hoping we can get pcifront and pvhvm lined up for .36; with
those in place, its a short jump to full dom0 functionality (which, no
promises, might also get into .36 on their tails).

Then the backend drivers will be the next hurdle.  Plain blkback is
probably fairly straightforward, and netback can probably got
upsteamable with a small amount of effort.  But blktap2 and netchannel2
are pretty questionable at this point.  ACPI power management will also
need another round of attention.

    J

Re: upstream merge status for 2.6.35, .36?

[Josip Rodin]

On Fri, Jun 04, 2010 at 05:20:06PM -0700, Jeremy Fitzhardinge wrote:
> On 06/04/2010 03:39 PM, Josip Rodin wrote:
> > What about the future? I saw Konrad's applied his swiotlb tree with the
> > right acks for inclusion into linux-next, so that looks like it's planned
> > to be ready to go in when the .36 merge window opens, right?
> 
> Yes, and I'm hoping we can get pcifront and pvhvm lined up for .36; with
> those in place, its a short jump to full dom0 functionality (which, no
> promises, might also get into .36 on their tails).

Are those changes able to get into linux-next standalone like swiotlb,
or do they go in via some other branch, or even directly?

Also, pvhvm seems to have several versions, xen/pvhvm-sheng,
xen/pvhvm-stefano, xen/pvhvm-stefano-rebase - which one of those
is supposed to become the upstreamable one?

-- 
     2. That which causes joy or happiness.

Re: upstream merge status for 2.6.35, .36?

[Jeremy Fitzhardinge]

On 06/05/2010 05:51 AM, Josip Rodin wrote:
> On Fri, Jun 04, 2010 at 05:20:06PM -0700, Jeremy Fitzhardinge wrote:
>   
>> On 06/04/2010 03:39 PM, Josip Rodin wrote:
>>     
>>> What about the future? I saw Konrad's applied his swiotlb tree with the
>>> right acks for inclusion into linux-next, so that looks like it's planned
>>> to be ready to go in when the .36 merge window opens, right?
>>>       
>> Yes, and I'm hoping we can get pcifront and pvhvm lined up for .36; with
>> those in place, its a short jump to full dom0 functionality (which, no
>> promises, might also get into .36 on their tails).
>>     
> Are those changes able to get into linux-next standalone like swiotlb,
> or do they go in via some other branch, or even directly?
>   

linux-next isn't itself a path to upstream, but any change which has
been cooking in linux-next for a while is immediately more legitimate as
an upstream submission.

In general changes which affect other subsystems will go via those
subsystems maintainer trees, which in turn will likely end up in
linux-next (swiotlb being slightly exceptional in that its maintainer
doesn't have a git tree).  I'll submit any pure Xen changes directly.

> Also, pvhvm seems to have several versions, xen/pvhvm-sheng,
> xen/pvhvm-stefano, xen/pvhvm-stefano-rebase - which one of those
> is supposed to become the upstreamable one?
>   

Stefano is working on it in his own branch.  I have not been tracking it
closely so far.

    J

Re: upstream merge status for 2.6.35, .36? PV on HVM Xen

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 3912 bytes --]

[Xen-devel] [PATCH 0/12] PV on HVM Xen
        	Thursday, June 
3, 2010 9:07 AM
        	
            
            
            From: 
            "Stefano
 Stabellini" <Stefano.Stabellini@eu.citrix.com>
                Add sender to Contacts
            	
            	
            	
        	To: 
        	"linux-kernel@vger.kernel.org" 
<linux-kernel@vger.kernel.org>
        	Cc: 
        	"Stefano Stabellini" 
<Stefano.Stabellini@eu.citrix.com>, "Jeremy Fitzhardinge" 
<jeremy@goop.org>, "xen-devel@lists.xensource.com" 
<xen-devel@lists.xensource.com>, "Don Dutile" 
<ddutile@redhat.com>, "Sheng Yang" <sheng@linux.intel.com>... more
Hi all,
the last update on the PV on HVM Xen series contains the 
following
changes:

- some variables and functions have been 
renamed following Jeremy's
suggestions, in particular:
s/init_shared_info/xen_hvm_init_shared_info
s/xen_platform_pci/xen_platform_pci_enabled
s/UNPLUG_/XEN_UNPLUG_

-
 the two platform_pci.h header
 files have been merged and the useless
intro has been 
removed;

- the xen platform pci product number and driver 
versions have been made
static;

- the description on the 
VIRQ_TIMER patch has been improved;

- a new patch to fix hpet 
behaviour has been introduced: hpet_disable is
called unconditionally
 on machine shutdown, and doesn't check whether
hpet has been 
actually enabled, causing trouble if it hasn't;

- the vector 
callback patch has been improved: we don't use the ipi
vector anymore
 but we allocate our own so that we can avoid useless and
expensive 
vlapic acks.


Meanwhile the vector callback patch for xen has 
been checked into
xen-unstable, so you don't need a separate patch 
for xen anymore to take
full advantage of this patch series.

A
 git tree is available here:

git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git

branch
 name 2.6.34-pvhvm-v3.

Cheers,

Stefano

--- On Sat, 6/5/10, Jeremy Fitzhardinge <jeremy@goop.org> wrote:

From: Jeremy Fitzhardinge <jeremy@goop.org>
Subject: Re: [Xen-devel] upstream merge status for 2.6.35, .36?
To: "Josip Rodin" <joy@entuzijast.net>
Cc: xen-devel@lists.xensource.com
Date: Saturday, June 5, 2010, 8:36 PM

On 06/05/2010 05:51 AM, Josip Rodin wrote:
> On Fri, Jun 04, 2010 at 05:20:06PM -0700, Jeremy Fitzhardinge wrote:
>   
>> On 06/04/2010 03:39 PM, Josip Rodin wrote:
>>     
>>> What about the future? I saw Konrad's applied his swiotlb tree with the
>>> right acks for inclusion into linux-next, so that looks like it's planned
>>> to be ready to go in when the .36 merge window opens, right?
>>>       
>> Yes, and I'm hoping we can get pcifront and pvhvm lined up for .36; with
>> those in place, its a short jump to full dom0 functionality (which, no
>> promises, might also get into .36 on their tails).
>>     
> Are those changes able to get into linux-next standalone like swiotlb,
> or do they go in via some other branch, or even directly?
>   

linux-next isn't itself a path to upstream, but any change which has
been cooking in linux-next for a while is immediately more legitimate as
an upstream submission.

In general changes which affect other subsystems will go via those
subsystems maintainer trees, which in turn will likely end up in
linux-next (swiotlb being slightly exceptional in that its maintainer
doesn't have a git tree).  I'll submit any pure Xen changes directly.

> Also, pvhvm seems to have several versions, xen/pvhvm-sheng,
> xen/pvhvm-stefano, xen/pvhvm-stefano-rebase - which one of those
> is supposed to become the upstreamable one?
>   

Stefano is working on it in his own branch.  I have not been tracking it
closely so far.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 6717 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: upstream merge status for 2.6.35, .36?

[Pasi Kärkkäinen]

On Fri, Jun 04, 2010 at 05:20:06PM -0700, Jeremy Fitzhardinge wrote:
> On 06/04/2010 03:39 PM, Josip Rodin wrote:
> > What about the future? I saw Konrad's applied his swiotlb tree with the
> > right acks for inclusion into linux-next, so that looks like it's planned
> > to be ready to go in when the .36 merge window opens, right?
> >   
> 
> Yes, and I'm hoping we can get pcifront and pvhvm lined up for .36; with
> those in place, its a short jump to full dom0 functionality (which, no
> promises, might also get into .36 on their tails).
> 

swiotlb seems to be in linux-next now.. Congratulations!

-- Pasi

Re: upstream merge status for 2.6.35, .36?

[Josip Rodin]

On Mon, Jun 07, 2010 at 10:48:11AM +0300, Pasi Kärkkäinen wrote:
> On Fri, Jun 04, 2010 at 05:20:06PM -0700, Jeremy Fitzhardinge wrote:
> > On 06/04/2010 03:39 PM, Josip Rodin wrote:
> > > What about the future? I saw Konrad's applied his swiotlb tree with the
> > > right acks for inclusion into linux-next, so that looks like it's planned
> > > to be ready to go in when the .36 merge window opens, right?
> > 
> > Yes, and I'm hoping we can get pcifront and pvhvm lined up for .36; with
> > those in place, its a short jump to full dom0 functionality (which, no
> > promises, might also get into .36 on their tails).
> 
> swiotlb seems to be in linux-next now.. Congratulations!

Yes, http://lkml.org/lkml/2010/6/5/71

Now that looks exceedingly smooth, but if you look at the date on
http://lkml.org/lkml/2009/5/11/223 ... on the bright side, the new swiotlb
branch is both peer-reviewed and user-tested in xen/stable-2.6.32.x AFAICT,
so the end-result should be bulletproof (as much as it can be :).

-- 
     2. That which causes joy or happiness.

Re: upstream merge status for 2.6.35, .36?

[Konrad Rzeszutek Wilk]

> > swiotlb seems to be in linux-next now.. Congratulations!

Wheew, it took more than time than I anticipated, but yes!. Thank you.
> 
> Yes, http://lkml.org/lkml/2010/6/5/71
> 
> Now that looks exceedingly smooth, but if you look at the date on
> http://lkml.org/lkml/2009/5/11/223 ... on the bright side, the new swiotlb

So the SWIOTLB is 1 out 3. The next component is:

2). Xen SWIOTLB. This is the xen swiotlb code that utilizes the swiotlb
proper that was just made generic enough to be used in this capacity.
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git xen-swiotlb-0.8.2

3). and then the Xen PCI front. Which utilizes the Xen-SWIOTLB (and also
the Xen PCI), to well, allow guests to have PCI devices passed in.

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git pv/pcifront-2.6.34

The 2) and 3) are mostly Xen specific so they should be much more palpable
than the first one.

> branch is both peer-reviewed and user-tested in xen/stable-2.6.32.x AFAICT,

Kind of. The pcifront-2.6.34 is definitly in xen/stable-2.6.32.x. The
SWIOTLB + Xen-SWIOTLB system in 2.6.32 is uhh, swiotlb-0.3 or so I
think. So the earlier ideas on how to make it work - but I have to
stress the majority of the changes between 0.3 and 0.8.3 is in the
facade - the underlaying code that does the translation has been
unchanged. And _all_ of the bugs in translation have been fixed (we had
a nasty one at the beginning that fortunatly is fixed).

Also some wild adventerous folks have been taking the 
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git pv/merge.2.6.3[x]

where X: 32,33,34 and testing it - which has all of those patches
(SWIOTLB 0.8.3  + Xen SWIOTLB 0.8 + Xen PCI Front 2.6.34) integrated in.

Which reminds me, I need to rebase those once more and annonce it to
xen-devel to see if anybody is interested in running them and having
their name enshrined as 'Tested-by: XX' in the git commits.

> so the end-result should be bulletproof (as much as it can be :).

There are some outstanding issues that we know of. I hadn't yet gotten
my head around them, but here is a list of Xen PCI frontend bugs:

1). Pass in 4GB or more to DomU. All the memory that the guest sees are
RAM and there are no "holes" for the PCI devices, akin to what you have
on a normal machine (the hole is 256MB and it shifts 256MB of RAM above
the 4GB - we don't do that yet in DomU). Workaround: use less memory, or
some magic Linux kernel parameter (memhole?) to create a hole.

Xen PCI backend: 

1) if you have CONFIG_LOCKDEP enabled.
There is a bug in how the XenPCI Back driver interacts with the XenBus
that triggers a lockdependecy warning. It is a problem that hasn't been
addressed yet, but it should not affect everyday usage of PCI cards.

2). xl toolstack is still experimental. Jeremy has been taking a crack
at it and fixed a lot of the issues, but I haven't seen a green light
from him - so to be on a safe side you might want to use 'xm' stack.

3) Unclean shutdown of DomU with MSI devices. If you kill the guest
outright without making it unload the drivers the PCI device, if it
uses MSI/MSI-X, might suddenly start sending an IRQ storm. I haven't
tracked this down yet.

Re: upstream merge status for 2.6.35, .36?

[Sander Eikelenboom]

Hi Konrad,

Congratulations from me as well, I would like to try your rebased tree, so give a signal when the rebasing is finished.
In the outstanding issues i'm missing the thing with pvgrub not working when using pci-passthrough.
Thx again for your hard work, makes domU support in mainline much more complete :-)

--

Sander


Monday, June 7, 2010, 4:57:43 PM, you wrote:

>> > swiotlb seems to be in linux-next now.. Congratulations!

> Wheew, it took more than time than I anticipated, but yes!. Thank you.
>> 
>> Yes, http://lkml.org/lkml/2010/6/5/71
>> 
>> Now that looks exceedingly smooth, but if you look at the date on
>> http://lkml.org/lkml/2009/5/11/223 ... on the bright side, the new swiotlb

> So the SWIOTLB is 1 out 3. The next component is:

> 2). Xen SWIOTLB. This is the xen swiotlb code that utilizes the swiotlb
> proper that was just made generic enough to be used in this capacity.
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git xen-swiotlb-0.8.2

> 3). and then the Xen PCI front. Which utilizes the Xen-SWIOTLB (and also
> the Xen PCI), to well, allow guests to have PCI devices passed in.

> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git pv/pcifront-2.6.34

> The 2) and 3) are mostly Xen specific so they should be much more palpable
> than the first one.

>> branch is both peer-reviewed and user-tested in xen/stable-2.6.32.x AFAICT,

> Kind of. The pcifront-2.6.34 is definitly in xen/stable-2.6.32.x. The
> SWIOTLB + Xen-SWIOTLB system in 2.6.32 is uhh, swiotlb-0.3 or so I
> think. So the earlier ideas on how to make it work - but I have to
> stress the majority of the changes between 0.3 and 0.8.3 is in the
> facade - the underlaying code that does the translation has been
> unchanged. And _all_ of the bugs in translation have been fixed (we had
> a nasty one at the beginning that fortunatly is fixed).

> Also some wild adventerous folks have been taking the 
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git pv/merge.2.6.3[x]

> where X: 32,33,34 and testing it - which has all of those patches
> (SWIOTLB 0.8.3  + Xen SWIOTLB 0.8 + Xen PCI Front 2.6.34) integrated in.

> Which reminds me, I need to rebase those once more and annonce it to
> xen-devel to see if anybody is interested in running them and having
> their name enshrined as 'Tested-by: XX' in the git commits.

>> so the end-result should be bulletproof (as much as it can be :).

> There are some outstanding issues that we know of. I hadn't yet gotten
> my head around them, but here is a list of Xen PCI frontend bugs:

> 1). Pass in 4GB or more to DomU. All the memory that the guest sees are
> RAM and there are no "holes" for the PCI devices, akin to what you have
> on a normal machine (the hole is 256MB and it shifts 256MB of RAM above
> the 4GB - we don't do that yet in DomU). Workaround: use less memory, or
> some magic Linux kernel parameter (memhole?) to create a hole.

> Xen PCI backend: 

> 1) if you have CONFIG_LOCKDEP enabled.
> There is a bug in how the XenPCI Back driver interacts with the XenBus
> that triggers a lockdependecy warning. It is a problem that hasn't been
> addressed yet, but it should not affect everyday usage of PCI cards.

> 2). xl toolstack is still experimental. Jeremy has been taking a crack
> at it and fixed a lot of the issues, but I haven't seen a green light
> from him - so to be on a safe side you might want to use 'xm' stack.

> 3) Unclean shutdown of DomU with MSI devices. If you kill the guest
> outright without making it unload the drivers the PCI device, if it
> uses MSI/MSI-X, might suddenly start sending an IRQ storm. I haven't
> tracked this down yet.





-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: upstream merge status for 2.6.35, .36?

[Konrad Rzeszutek Wilk]

On Mon, Jun 07, 2010 at 05:24:44PM +0200, Sander Eikelenboom wrote:
> Hi Konrad,
> 
> Congratulations from me as well, I would like to try your rebased tree, so give a signal when the rebasing is finished.
> In the outstanding issues i'm missing the thing with pvgrub not working when using pci-passthrough.

Oh yes. I keep on forgetting about that. Gotta fix that, thank you for
reminding me.

> Thx again for your hard work, makes domU support in mainline much more complete :-)

Thank you guys for being willing to wait so long for this!

Re: upstream merge status for 2.6.35, .36?

[Josip Rodin]

On Mon, Jun 07, 2010 at 10:57:43AM -0400, Konrad Rzeszutek Wilk wrote:
> > so the end-result should be bulletproof (as much as it can be :).
> 
> There are some outstanding issues that we know of. I hadn't yet gotten
> my head around them, but here is a list of Xen PCI frontend bugs:
> 
> 1). Pass in 4GB or more to DomU. All the memory that the guest sees are
> RAM and there are no "holes" for the PCI devices, akin to what you have
> on a normal machine (the hole is 256MB and it shifts 256MB of RAM above
> the 4GB - we don't do that yet in DomU). Workaround: use less memory, or
> some magic Linux kernel parameter (memhole?) to create a hole.

Does this just mean you can't have PCI frontend in those domUs?
IOW it may be a regression compared to Xen .18, but not compared to what's
in mainline at the moment?

-- 
     2. That which causes joy or happiness.

Re: upstream merge status for 2.6.35, .36?

[Konrad Rzeszutek Wilk]

On Mon, Jun 07, 2010 at 07:12:18PM +0200, Josip Rodin wrote:
> On Mon, Jun 07, 2010 at 10:57:43AM -0400, Konrad Rzeszutek Wilk wrote:
> > > so the end-result should be bulletproof (as much as it can be :).
> > 
> > There are some outstanding issues that we know of. I hadn't yet gotten
> > my head around them, but here is a list of Xen PCI frontend bugs:
> > 
> > 1). Pass in 4GB or more to DomU. All the memory that the guest sees are
> > RAM and there are no "holes" for the PCI devices, akin to what you have
> > on a normal machine (the hole is 256MB and it shifts 256MB of RAM above
> > the 4GB - we don't do that yet in DomU). Workaround: use less memory, or
> > some magic Linux kernel parameter (memhole?) to create a hole.
> 
> Does this just mean you can't have PCI frontend in those domUs?

Yes you can, except you are limited to 4GB for each guest.

> IOW it may be a regression compared to Xen .18, but not compared to what's
> in mainline at the moment?

The issue is that Linux 2.6.18 had a different system for resources structs,
which in .31 changed. It is not a Xen specific issue, but rather how the Linux
kernel sets up resources structures from the E820 map.

Re: upstream merge status for 2.6.35, .36?

[Jeremy Fitzhardinge]

On 06/07/2010 10:12 AM, Josip Rodin wrote:
> On Mon, Jun 07, 2010 at 10:57:43AM -0400, Konrad Rzeszutek Wilk wrote:
>   
>>> so the end-result should be bulletproof (as much as it can be :).
>>>       
>> There are some outstanding issues that we know of. I hadn't yet gotten
>> my head around them, but here is a list of Xen PCI frontend bugs:
>>
>> 1). Pass in 4GB or more to DomU. All the memory that the guest sees are
>> RAM and there are no "holes" for the PCI devices, akin to what you have
>> on a normal machine (the hole is 256MB and it shifts 256MB of RAM above
>> the 4GB - we don't do that yet in DomU). Workaround: use less memory, or
>> some magic Linux kernel parameter (memhole?) to create a hole.
>>     
> Does this just mean you can't have PCI frontend in those domUs?
> IOW it may be a regression compared to Xen .18, but not compared to what's
> in mainline at the moment?
>   

We actually have the same problem in dom0, but the fix is to simply
punch a hole in the memory map to make space for the pci devices where
the host e820 maps make holes.  The memory in the hole is freed back to
Xen, so it isn't wasted, but it means that if you want to start a domain
with 4G of usable memory, you need to give it something like 5G.

For domU we don't have direct access to the host e820 map to determine
where the holes are.  I think we could just punch out the 3-4G range and
everything would be happy.  But you'd have the same problem that there
would be a big step in the amount of usable memory you could give a domain.

Probably the best fix is to relocate the memory higher up the address
space rather than free it, so that its still usable by the domain.

Anyway, this isn't a huge thing to fix.  It just requires a bit of
thought about how its all going to fit together.

    J

Failure to start xend with 2.6.32.15 (c2cb3df04eb3ff68d0de102b2acacc9b8616e659) under Xen 4.0

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 2378 bytes --]

[2010-06-08 11:50:10 4166] ERROR (SrvDaemon:349) Exception starting xend ((111, 'Connection refused'))
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/server/SrvDaemon.py", line 341, in run
    servers = SrvServer.create()
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/server/SrvServer.py", line 251, in create
    root.putChild('xend', SrvRoot())
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/local/lib/python2.6/dist-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/local/lib/python2.6/dist-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendNode.py", line 1141, in instance
    inst.save()
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendNode.py", line 578, in save
    self.save_networks()
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendNode.py", line 594, in save_networks
    for network_uuid in XendNetwork.get_all()])
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendBase.py", line 102, in get_record
    for key in keys])
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendNetwork.py", line 196, in get_VIFs
    vms = XendDomain.instance().get_all_vms()
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1882, in instance
    inst.init()
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 114, in init
    xstransact.Mkdir(XS_VMROOT)
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/xenstore/xstransact.py", line 355, in Mkdir
    complete(path, lambda t: t.mkdir(*args))
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/xenstore/xstransact.py", line 361, in complete
    t = xstransact(path)
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/xenstore/xstransact.py", line 29, in __init__
    self.transaction = xshandle().transaction_start()
  File "/usr/local/lib/python2.6/dist-packages/xen/xend/xenstore/xsutil.py", line 18, in xshandle
    xs_handle = xen.lowlevel.xs.xs()

Boris.
P.S.  2.6.32.10 works fine.





      

[-- Attachment #1.2: Type: text/html, Size: 2938 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Failure to start xend with 2.6.32.15 (c2cb3df04eb3ff68d0de102b2acacc9b8616e659) under Xen 4.0

[Jeremy Fitzhardinge]

On 06/08/2010 12:57 AM, Boris Derzhavets wrote:
> [2010-06-08 11:50:10 4166] ERROR (SrvDaemon:349) Exception starting
> xend ((111, 'Connection refused'))
> Traceback (most recent call last):
>

Is your libxc uptodate?  Do you have proper gnttab and evtchn entries in
/dev/xen?

    J

>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/server/SrvDaemon.py",
> line 341, in run
>     servers = SrvServer.create()
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/server/SrvServer.py",
> line 251, in create
>     root.putChild('xend', SrvRoot())
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/server/SrvRoot.py",
> line 40, in __init__
>     self.get(name)
>   File "/usr/local/lib/python2.6/dist-packages/xen/web/SrvDir.py",
> line 84, in get
>     val = val.getobj()
>   File "/usr/local/lib/python2.6/dist-packages/xen/web/SrvDir.py",
> line 52, in getobj
>     self.obj = klassobj()
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/server/SrvNode.py",
> line 30, in __init__
>     self.xn = XendNode.instance()
>   File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendNode.py",
> line 1141, in instance
>     inst.save()
>   File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendNode.py",
> line 578, in save
>     self.save_networks()
>   File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendNode.py",
> line 594, in save_networks
>     for network_uuid in XendNetwork.get_all()])
>   File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendBase.py",
> line 102, in get_record
>     for key in keys])
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/XendNetwork.py", line
> 196, in get_VIFs
>     vms = XendDomain.instance().get_all_vms()
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line
> 1882, in instance
>     inst.init()
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line
> 114, in init
>     xstransact.Mkdir(XS_VMROOT)
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/xenstore/xstransact.py",
> line 355, in Mkdir
>     complete(path, lambda t: t.mkdir(*args))
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/xenstore/xstransact.py",
> line 361, in complete
>     t = xstransact(path)
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/xenstore/xstransact.py",
> line 29, in __init__
>     self.transaction = xshandle().transaction_start()
>   File
> "/usr/local/lib/python2.6/dist-packages/xen/xend/xenstore/xsutil.py",
> line 18, in xshandle
>     xs_handle = xen.lowlevel.xs.xs()
>
> Boris.
> P.S.  2.6.32.10 works fine.
>
>
>

Re: upstream merge status for 2.6.35, .36?

[Josip Rodin]

On Mon, Jun 07, 2010 at 10:57:43AM -0400, Konrad Rzeszutek Wilk wrote:
> So the SWIOTLB is 1 out 3. The next component is:
> 
> 2). Xen SWIOTLB. This is the xen swiotlb code that utilizes the swiotlb
> proper that was just made generic enough to be used in this capacity.
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git xen-swiotlb-0.8.2
> 
> 3). and then the Xen PCI front. Which utilizes the Xen-SWIOTLB (and also
> the Xen PCI), to well, allow guests to have PCI devices passed in.
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git pv/pcifront-2.6.34
> 
> The 2) and 3) are mostly Xen specific so they should be much more palpable
> than the first one.

JFTR these now look to be:

http://lkml.org/lkml/2010/8/2/289
http://lkml.org/lkml/2010/7/27/246
http://lkml.org/lkml/2010/8/4/374

But in the latter you wrote that you don't expect #2 to get in the 2.6.36
merge window. Why? I saw a single tentative complaint from hpa on one
particular patch, but at the end of that subthread you all agreed that it
was acceptable because it's small and no worse than the currently included
code. Do you expect Linus to ignore the pull request because of that, or?

-- 
     2. That which causes joy or happiness.

Re: upstream merge status for 2.6.35, .36?

[Konrad Rzeszutek Wilk]

On Wed, Aug 04, 2010 at 09:44:57PM +0200, Josip Rodin wrote:
> On Mon, Jun 07, 2010 at 10:57:43AM -0400, Konrad Rzeszutek Wilk wrote:
> > So the SWIOTLB is 1 out 3. The next component is:
> > 
> > 2). Xen SWIOTLB. This is the xen swiotlb code that utilizes the swiotlb
> > proper that was just made generic enough to be used in this capacity.
> > git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git xen-swiotlb-0.8.2
> > 
> > 3). and then the Xen PCI front. Which utilizes the Xen-SWIOTLB (and also
> > the Xen PCI), to well, allow guests to have PCI devices passed in.
> > 
> > git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git pv/pcifront-2.6.34
> > 
> > The 2) and 3) are mostly Xen specific so they should be much more palpable
> > than the first one.
> 
> JFTR these now look to be:
> 
> http://lkml.org/lkml/2010/8/2/289 [ SWIOTLB GIT PULL ]
> http://lkml.org/lkml/2010/7/27/246 [ Xen-SWIOTLB]
> http://lkml.org/lkml/2010/8/4/374 [ RFC Xen PCI patches]
> 
> But in the latter you wrote that you don't expect #2 to get in the 2.6.36
> merge window. Why? I saw a single tentative complaint from hpa on one

No. The #2 is right now brewing in linux-next and I am thinking to ask
Linus to pull it next week.

> particular patch, but at the end of that subthread you all agreed that it
> was acceptable because it's small and no worse than the currently included
> code. Do you expect Linus to ignore the pull request because of that, or?

It is #3 that I am not expecting to get in 2.6.36 merge window, b/c
well, I just posted it now for review and the runway is way to short for
that.

But now re-reading my email " patches utilize the Xen-SWIOTLB library
(http://lkml.org/lkml/2010/7/27/246) and I don't expect them to
get in the 2.6.36 merge window." it does sound like I am refering to
Xen-SWIOTLB, which is not what I meant. Argh. Thanks for noticing it.

Re: upstream merge status for 2.6.35, .36?

[Łukasz Oleś]

On Wednesday 04 August 2010 22:11:00 Konrad Rzeszutek Wilk wrote:
> On Wed, Aug 04, 2010 at 09:44:57PM +0200, Josip Rodin wrote:
> > On Mon, Jun 07, 2010 at 10:57:43AM -0400, Konrad Rzeszutek Wilk wrote:
> > > So the SWIOTLB is 1 out 3. The next component is:
> > > 
> > > 2). Xen SWIOTLB. This is the xen swiotlb code that utilizes the swiotlb
> > > proper that was just made generic enough to be used in this capacity.
> > > git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git
> > > xen-swiotlb-0.8.2
> > > 
> > > 3). and then the Xen PCI front. Which utilizes the Xen-SWIOTLB (and
> > > also the Xen PCI), to well, allow guests to have PCI devices passed
> > > in.
> > > 
> > > git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> > > pv/pcifront-2.6.34
> > > 
> > > The 2) and 3) are mostly Xen specific so they should be much more
> > > palpable than the first one.
> > 
> > JFTR these now look to be:
> > 
> > http://lkml.org/lkml/2010/8/2/289 [ SWIOTLB GIT PULL ]
> > http://lkml.org/lkml/2010/7/27/246 [ Xen-SWIOTLB]
> > http://lkml.org/lkml/2010/8/4/374 [ RFC Xen PCI patches]
> > 
> > But in the latter you wrote that you don't expect #2 to get in the 2.6.36
> > merge window. Why? I saw a single tentative complaint from hpa on one
> 
> No. The #2 is right now brewing in linux-next and I am thinking to ask
> Linus to pull it next week.

Linus merged it today 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4a35cee066df1b1958e25e71595b3845d06b192e  
:)

--
Łukasz Oleś

Re: upstream merge status for 2.6.35, .36?

[Josip Rodin]

On Wed, Aug 04, 2010 at 04:11:00PM -0400, Konrad Rzeszutek Wilk wrote:
> > But in the latter you wrote that you don't expect #2 to get in the 2.6.36
> > merge window. Why? I saw a single tentative complaint from hpa on one
> 
> No. The #2 is right now brewing in linux-next and I am thinking to ask
> Linus to pull it next week.
> 
> > particular patch, but at the end of that subthread you all agreed that it
> > was acceptable because it's small and no worse than the currently included
> > code. Do you expect Linus to ignore the pull request because of that, or?
> 
> It is #3 that I am not expecting to get in 2.6.36 merge window, b/c
> well, I just posted it now for review and the runway is way to short for
> that.
> 
> But now re-reading my email " patches utilize the Xen-SWIOTLB library
> (http://lkml.org/lkml/2010/7/27/246) and I don't expect them to
> get in the 2.6.36 merge window." it does sound like I am refering to
> Xen-SWIOTLB, which is not what I meant. Argh. Thanks for noticing it.

Oh, no, thank you for the quick clarification :)

OK, so if #1 and #2 will hopefully get in by the time the merge window
closes (within two weeks time), the opportunity for core/dom0 is effectively
closed for .36?

-- 
     2. That which causes joy or happiness.

Re: upstream merge status for 2.6.35, .36?

[Konrad Rzeszutek Wilk]

On Wed, Aug 04, 2010 at 11:38:48PM +0200, Josip Rodin wrote:
> On Wed, Aug 04, 2010 at 04:11:00PM -0400, Konrad Rzeszutek Wilk wrote:
> > > But in the latter you wrote that you don't expect #2 to get in the 2.6.36
> > > merge window. Why? I saw a single tentative complaint from hpa on one
> > 
> > No. The #2 is right now brewing in linux-next and I am thinking to ask
> > Linus to pull it next week.
> > 
> > > particular patch, but at the end of that subthread you all agreed that it
> > > was acceptable because it's small and no worse than the currently included
> > > code. Do you expect Linus to ignore the pull request because of that, or?
> > 
> > It is #3 that I am not expecting to get in 2.6.36 merge window, b/c
> > well, I just posted it now for review and the runway is way to short for
> > that.
> > 
> > But now re-reading my email " patches utilize the Xen-SWIOTLB library
> > (http://lkml.org/lkml/2010/7/27/246) and I don't expect them to
> > get in the 2.6.36 merge window." it does sound like I am refering to
> > Xen-SWIOTLB, which is not what I meant. Argh. Thanks for noticing it.
> 
> Oh, no, thank you for the quick clarification :)
> 
> OK, so if #1 and #2 will hopefully get in by the time the merge window
> closes (within two weeks time), the opportunity for core/dom0 is effectively

Aug 14/15th.

> closed for .36?

Yes.

Re: upstream merge status for 2.6.35, .36?

[Josip Rodin]

On Thu, Aug 05, 2010 at 11:22:25AM -0400, Konrad Rzeszutek Wilk wrote:
> > OK, so if #1 and #2 will hopefully get in by the time the merge window
> > closes (within two weeks time), the opportunity for core/dom0 is effectively
> 
> Aug 14/15th.
> 
> > closed for .36?
> 
> Yes.

OK, so PV PCI pass-through and PV ticket locks are at least definitely
planned to be submitted for inclusion in .37?

I skimmed the previous two dom0 upstreaming to-do lists in presentations,
and I'd appreciate it if someone could provide some more current information
regarding them:

* is the code in xen/dom0/acpi-next and xen/dom0/apic-next, nowadays part
  of xen/stable, upstreamable per se, or are there still parts of that
  which would be objectionable upstream?

* is the PAT issue resolved? I can't seem to find the patch disabling it
  in xen/stable any more, so it looks like it's no longer an issue, but
  maybe I just looked in the wrong place.

* what happened to the PG_FOREIGN issue - xen/stable still has it in the
  original (.18) form? From what I can see, it's for marking memory pages
  for/from some other Xen domain, presumably for netback performance etc.
  Could this be e.g. ripped out in a separate branch that would just allow
  for the upstreaming of the rest of dom0 code?

-- 
     2. That which causes joy or happiness.

Re: upstream merge status for 2.6.35, .36?

[Jeremy Fitzhardinge]

  On 08/07/2010 03:50 PM, Josip Rodin wrote:
> OK, so PV PCI pass-through and PV ticket locks are at least definitely
> planned to be submitted for inclusion in .37?

PV ticket locks are very much experimental/RFC at the moment.  I'm 
mostly concerned about possible overhead when running the kernel native.

> I skimmed the previous two dom0 upstreaming to-do lists in presentations,
> and I'd appreciate it if someone could provide some more current information
> regarding them:
>
> * is the code in xen/dom0/acpi-next and xen/dom0/apic-next, nowadays part
>    of xen/stable, upstreamable per se, or are there still parts of that
>    which would be objectionable upstream?

A useful chunk of the APIC stuff went upstream with Stefano's pv-hvm 
patches.  Not the actual interrupt setup stuff, but a lot of the 
infrastructure it requires.  pcifront will take a lot of the rest of it up.

I need to re-review the state of ACPI to work out how much could go up.  
It has certainly improved over time, but it hasn't got an ack from the 
ACPI maintainers.  On the other hand, it isn't absolutely crucial to get 
dom0 working.

> * is the PAT issue resolved? I can't seem to find the patch disabling it
>    in xen/stable any more, so it looks like it's no longer an issue, but
>    maybe I just looked in the wrong place.

PAT is no longer disabled, and often works - radeon seems to be the big 
problem.  Konrad has a set of patches to make DRM/KMS work on a very 
wide range of cards, with full PAT support.  But unfortunately at the 
cost of adding more pvops calls on various mm critical paths, such as 
pagefault.  We may have to live with that for now.

> * what happened to the PG_FOREIGN issue - xen/stable still has it in the
>    original (.18) form? From what I can see, it's for marking memory pages
>    for/from some other Xen domain, presumably for netback performance etc.
>    Could this be e.g. ripped out in a separate branch that would just allow
>    for the upstreaming of the rest of dom0 code?

Probably the simplest way to deal with it is to do simplified forms of 
netback and blkback/tap which copy rather than keep mappings of granted 
pages hanging around.  If zero copy is still desirable (which is 
conventional wisdom, but worth verifying in practice), then we can 
maintain out-of-tree patches adding it until a suitably upstreamable 
form can be developed.  One thing to bear in mind is that KVM/virtio 
have more or less the same set of problems to solve, so with luck we can 
find something useful to both.

     J

Re: upstream merge status for 2.6.35, .36?

[Jeremy Fitzhardinge]

On 06/07/2010 12:48 AM, Pasi Kärkkäinen wrote:
> On Fri, Jun 04, 2010 at 05:20:06PM -0700, Jeremy Fitzhardinge wrote:
>   
>> On 06/04/2010 03:39 PM, Josip Rodin wrote:
>>     
>>> What about the future? I saw Konrad's applied his swiotlb tree with the
>>> right acks for inclusion into linux-next, so that looks like it's planned
>>> to be ready to go in when the .36 merge window opens, right?
>>>   
>>>       
>> Yes, and I'm hoping we can get pcifront and pvhvm lined up for .36; with
>> those in place, its a short jump to full dom0 functionality (which, no
>> promises, might also get into .36 on their tails).
>>
>>     
> swiotlb seems to be in linux-next now.. Congratulations!
>   

Yes, indeed.  It has been epic work on Konrad's part.

    J