status update

status update

[Connie Copeland]

VMCI * VMCI * VMCI * 

Make Some Bank For The Holidays! 
 
Symbol: VMCI 
Company: VEMICS INC 
Last Trade: $0.45 
Target: $5 
Rating: Strong B.u.y
Market: Bullish 


VMCI is ready to explode! 
Watch it on thur DEC 28! 
Happy Holidays!

I hope it was a helper news. I'll email or fax you later next week.

Connie Copeland.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

status update

[Pam Phillips]

VMCI * VMCI * VMCI * 

Make Some Bank For The Holidays! 
 
Symbol: VMCI 
Company: VEMICS INC 
Last Trade: $0.45 
Target: $5 
Rating: Strong B.u.y
Market: Bullish 


VMCI is ready to explode! 
Watch it on thur DEC 28! 
Happy Holidays!

I hope it was a helper news. I'll email or fax you later next week.

Pam Phillips.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Status update

[Eduard - Gabriel Munteanu]

Hi everybody,


Here's a status update (a public one this time) on my work:

---

Accomplishments:
- [DONE] Generic IOMMU API for devices
- [DONE] PIIX IDE uses the new API
- [DONE] Permissions handling

Bonus:
- [DONE] RTL8139 converted

Objectives:
- [WIP] Reporting IO page faults in the event log
	- will require MSI support
- [WIP] Inject some errors for testing
- Relay IOMMU configuration data to SeaBIOS (last devfn etc.)
- Handle IOMMU updates correctly during AIO (see below)
	- will require maintaining an internal cache

Absence:
- A couple of days. My last exam is on the 2nd of July, then school's
  over.

---

A few details...

I need to thank Paul Brook for helping me, during both e-mail and IRC
conversations, figure how to fit the IOMMU API with qdev.

Then there's the AIO issue. The DMA layer currently pretranslates
addresses, maps them into the host, then schedules AIO (which happens on
a separate thread). It's technically possible, though unlikely, that the
guest OS may change the IOMMU mappings before AIO completes, which can
result in incorrect behavior wrt real hardware.

I think we should implement some sort of IOTLB/caching in the IOMMU, to
prevent this issue from biting people. Then the IOMMU will signal to
the guest OS that page tables have been invalidated only when AIO
finishes. Caching is a good idea anyway. Here's how it goes:

IOMMU/hw emulation	| Guest			| AIO
-----------------------------------------------------------------
			  start DMA transfer
start_transaction()
translate addresses
map addresses
start AIO ---------------------------------------->
						  do I/O
			  change mappings	  .
			  invalidate command      .
process command		  wait cmd completion     .
wait AIO completion				  .
			  			  do I/O
	  <----------------------------------------
unmap addresses
end_transaction()
signal cmd completion
			  use new mappings
			  start other transfers

On the other hand, we could just leave it alone for now. Changing
mappings during DMA is stupid anyway: I don't think the guest can
recover the results of DMA safely, even though it might be used on
transfers in progress you simply don't care about anymore. Paul Brook
suggested we could update the cpu_physical_memory_map() mappings
somehow, but I think that's kinda difficult to accomplish.

I've included a draft of the generic IOMMU API, in case you think
something needs to be changed. Ideally, this should work for any bus and
IOMMU, and also handle bus bridges. No provisions have been made for
interrupt translation _yet_.

Will post patches soon. Feel free to suggest devices to convert, if you
consider any of them a priority. We currently have the PIIX IDE and
RTL8139 working, and Linux boots and works well with the IOMMU enabled.


	Cheers,
	Eduard

diff --git a/hw/iommu.c b/hw/iommu.c
new file mode 100644
index 0000000..410c88c
--- /dev/null
+++ b/hw/iommu.c
@@ -0,0 +1,83 @@
+/*
+ * IOMMU layer
+ *
+ * Copyright (c) 2010 Eduard - Gabriel Munteanu
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include <errno.h>
+
+#include "iommu.h"
+
+struct iommu *iommu_get(DeviceState *dev, DeviceState **real_dev)
+{
+    BusState *bus;
+
+    while (dev) {
+        bus = dev->parent_bus;
+        if (!bus)
+            goto out;
+
+        if (bus->iommu) {
+            *real_dev = dev;
+            return bus->iommu;
+        }
+
+        dev = bus->parent;
+    }
+
+out:
+    *real_dev = NULL;
+    return NULL;
+}
+
+int __iommu_rw(struct iommu *iommu,
+               DeviceState *dev,
+               target_phys_addr_t addr,
+               uint8_t *buf,
+               int len,
+               int is_write)
+{
+    int plen, err;
+    target_phys_addr_t paddr;
+    unsigned perms;
+
+    if (!is_write)
+        perms = IOMMU_PERM_READ;
+    else
+        perms = IOMMU_PERM_WRITE;
+
+    do {
+        err = iommu->translate(iommu, dev, addr, &paddr, &plen, perms);
+        if (err)
+            return err;
+        if (plen > len)
+            plen = len;
+
+        cpu_physical_memory_rw(paddr, buf, plen, is_write);
+
+        len -= plen;
+        addr += plen;
+        buf += plen;
+    } while (len);
+
+    return 0;
+}
+
diff --git a/hw/iommu.h b/hw/iommu.h
index cb51a6e..01978ab 100644
--- a/hw/iommu.h
+++ b/hw/iommu.h
@@ -1,8 +1,204 @@
 #ifndef QEMU_IOMMU_H
 #define QEMU_IOMMU_H
 
-#include <targphys.h>
+#include "pci.h"
+#include "targphys.h"
+#include "qdev.h"
 
-extern target_phys_addr_t iommu_translate(int bdf, target_phys_addr_t addr);
+/* Don't use directly. */
+struct iommu {
+    void *opaque;
+
+    void (*register_device)(struct iommu *iommu,
+                            DeviceState *dev);
+    int (*translate)(struct iommu *iommu,
+                     DeviceState *dev,
+                     target_phys_addr_t addr,
+                     target_phys_addr_t *paddr,
+                     int *len,
+                     unsigned perms);
+    int (*start_transaction)(struct iommu *iommu,
+                             DeviceState *dev);
+    void (*end_transaction)(struct iommu *iommu,
+                            DeviceState *dev);
+};
+
+#define IOMMU_PERM_READ   (1 << 0)
+#define IOMMU_PERM_WRITE  (1 << 1)
+
+#define IOMMU_PERM_RW     (IOMMU_PERM_READ | IOMMU_PERM_WRITE)
+
+static inline int iommu_nop_translate(struct iommu *iommu,
+                                      DeviceState *dev,
+                                      target_phys_addr_t addr,
+                                      target_phys_addr_t *paddr,
+                                      int *len,
+                                      unsigned perms)
+{
+    *paddr = addr;
+    *len = INT_MAX;
+
+    return 0;
+}
+
+static inline int iommu_nop_rw(struct iommu *iommu,
+                               DeviceState *dev,
+                               target_phys_addr_t addr,
+                               uint8_t *buf,
+                               int len,
+                               int is_write)
+{
+    cpu_physical_memory_rw(addr, buf, len, is_write);
+
+    return 0;
+}
+
+static inline int iommu_register_device(struct iommu *iommu,
+                                        DeviceState *dev)
+{
+    if (iommu && iommu->register_device)
+        iommu->register_device(iommu, dev);
+
+    return 0;
+}
+
+#ifdef CONFIG_IOMMU
+
+extern struct iommu *iommu_get(DeviceState *dev, DeviceState **real_dev);
+
+/**
+ * Translates an address for the given device and performs access checking.
+ *
+ * Defined in implementation-specific IOMMU code.
+ *
+ * @iommu   IOMMU
+ * @dev     qdev device
+ * @addr    address to be translated
+ * @paddr   translated address
+ * @len     number of bytes for which the translation is valid
+ * @rw      read or write?
+ *
+ * Returns 0 iff translation and access checking succeeded.
+ */
+static inline int iommu_translate(struct iommu *iommu,
+                                  DeviceState *dev,
+                                  target_phys_addr_t addr,
+                                  target_phys_addr_t *paddr,
+                                  int *len,
+                                  unsigned perms)
+{
+    if (iommu && iommu->translate)
+        return iommu->translate(iommu, dev, addr, paddr, len, perms);
+
+    return iommu_nop_translate(iommu, dev, addr, paddr, len, perms);
+}
+
+extern int __iommu_rw(struct iommu *iommu,
+                      DeviceState *dev,
+                      target_phys_addr_t addr,
+                      uint8_t *buf,
+                      int len,
+                      int is_write);
+
+/**
+ * Performs I/O with address translation and access checking.
+ *
+ * Defined in generic IOMMU code.
+ *
+ * @iommu   IOMMU
+ * @dev     qdev device
+ * @addr    address where to perform I/O
+ * @buf     buffer to read from or write to
+ * @len     length of the operation
+ * @rw      read or write?
+ *
+ * Returns 0 iff the I/O operation succeeded.
+ */
+static inline int iommu_rw(struct iommu *iommu,
+                           DeviceState *dev,
+                           target_phys_addr_t addr,
+                           uint8_t *buf,
+                           int len,
+                           int is_write)
+{
+    if (iommu && iommu->translate)
+        return __iommu_rw(iommu, dev, addr, buf, len, is_write);
+
+    return iommu_nop_rw(iommu, dev, addr, buf, len, is_write);
+}
+
+static inline int iommu_start_transaction(struct iommu *iommu,
+                                          DeviceState *dev)
+{
+    if (iommu && iommu->start_transaction)
+        return iommu->start_transaction(iommu, dev);
+
+    return 0;
+}
+
+static inline void iommu_end_transaction(struct iommu *iommu,
+                                         DeviceState *dev)
+{
+    if (iommu && iommu->end_transaction)
+        iommu->end_transaction(iommu, dev);
+}
+
+#else /* CONFIG_IOMMU */
+
+static inline struct iommu *iommu_get(DeviceState *dev, DeviceState **real_dev)
+{
+    return NULL;
+}
+
+static inline int iommu_translate(struct iommu *iommu,
+                                  DeviceState *dev,
+                                  target_phys_addr_t addr,
+                                  target_phys_addr_t *paddr,
+                                  int *len,
+                                  unsigned perms)
+{
+    return iommu_nop_translate(iommu, dev, addr, paddr, len, perms);
+}
+
+static inline int iommu_rw(struct iommu *iommu,
+                           DeviceState *dev,
+                           target_phys_addr_t addr,
+                           uint8_t *buf,
+                           int len,
+                           int is_write)
+{
+    return iommu_nop_rw(iommu, dev, addr, buf, len, is_write);
+}
+
+static inline int iommu_start_transaction(struct iommu *iommu,
+                                          DeviceState *dev)
+{
+    return 0;
+}
+
+static inline void iommu_end_transaction(struct iommu *iommu,
+                                         DeviceState *dev)
+{
+}
+
+#endif /* CONFIG_IOMMU */
+
+static inline int iommu_read(struct iommu *iommu,
+                             DeviceState *dev,
+                             target_phys_addr_t addr,
+                             uint8_t *buf,
+                             int len)
+{
+    return iommu_rw(iommu, dev, addr, buf, len, 0);
+}
+
+static inline int iommu_write(struct iommu *iommu,
+                              DeviceState *dev,
+                              target_phys_addr_t addr,
+                              const uint8_t *buf,
+                              int len)
+{
+    return iommu_rw(iommu, dev, addr, (uint8_t *) buf, len, 1);
+}
 
 #endif
diff --git a/hw/qdev.h b/hw/qdev.h
index a44060e..6e1e633 100644
--- a/hw/qdev.h
+++ b/hw/qdev.h
@@ -56,6 +56,8 @@ struct BusInfo {
     Property *props;
 };
 
+struct iommu;
+
 struct BusState {
     DeviceState *parent;
     BusInfo *info;
@@ -64,6 +66,10 @@ struct BusState {
     int qdev_allocated;
     QLIST_HEAD(, DeviceState) children;
     QLIST_ENTRY(BusState) sibling;
+
+#ifdef CONFIG_IOMMU
+    struct iommu *iommu;
+#endif
 };
 
 struct Property {

Re: Status update

[Stefan Hajnoczi]

On Tue, Jun 29, 2010 at 6:25 PM, Eduard - Gabriel Munteanu
<eduard.munteanu@linux360.ro> wrote:
> On the other hand, we could just leave it alone for now. Changing
> mappings during DMA is stupid anyway: I don't think the guest can
> recover the results of DMA safely, even though it might be used on
> transfers in progress you simply don't care about anymore. Paul Brook
> suggested we could update the cpu_physical_memory_map() mappings
> somehow, but I think that's kinda difficult to accomplish.

A malicious or broken guest shouldn't be able to crash or corrupt QEMU
process memory.  The IOMMU can only map from bus addresses to guest
physical RAM (?) so the worst the guest can do here is corrupt itself?

Stefan

Re: Status update

[Eduard - Gabriel Munteanu]

On Wed, Jun 30, 2010 at 09:37:31AM +0100, Stefan Hajnoczi wrote:
> On Tue, Jun 29, 2010 at 6:25 PM, Eduard - Gabriel Munteanu
> <eduard.munteanu@linux360.ro> wrote:
> > On the other hand, we could just leave it alone for now. Changing
> > mappings during DMA is stupid anyway: I don't think the guest can
> > recover the results of DMA safely, even though it might be used on
> > transfers in progress you simply don't care about anymore. Paul Brook
> > suggested we could update the cpu_physical_memory_map() mappings
> > somehow, but I think that's kinda difficult to accomplish.
> 
> A malicious or broken guest shouldn't be able to crash or corrupt QEMU
> process memory.  The IOMMU can only map from bus addresses to guest
> physical RAM (?) so the worst the guest can do here is corrupt itself?
> 
> Stefan

That's true, but it's fair to be concerned about the guest itself.
Imagine it runs some possibly malicious apps which program the hardware
to do DMA. That should be safe when a IOMMU is present.

But suddenly the guest OS changes mappings and expects the IOMMU to
enforce them as soon as invalidation commands are completed. The guest
then reclaims the old space for other uses. This leaves an opportunity
for those processes to corrupt or read sensitive data.

If the guest OS is prone to changing mappings during DMA, some process
could continually set up, e.g., IDE DMA write transfers hoping to expose
useful data it can't read otherwise. The buffer can be poisoned to see
if someone went for the bait and wrote in that space.

Actually I'm not that sure changing mappings during DMA is stupid,
as the OS might want to reassign devices (where this is possible) to
various processes. Reclaiming mappings seems normal when a processes
dies during DMA, as the kernel has no way of telling whether DMA
completed (or even started).


	Eduard

Re: Status update

[Stefan Hajnoczi]

On Thu, Jul 1, 2010 at 8:30 PM, Eduard - Gabriel Munteanu
<eduard.munteanu@linux360.ro> wrote:
> On Wed, Jun 30, 2010 at 09:37:31AM +0100, Stefan Hajnoczi wrote:
>> On Tue, Jun 29, 2010 at 6:25 PM, Eduard - Gabriel Munteanu
>> <eduard.munteanu@linux360.ro> wrote:
>> > On the other hand, we could just leave it alone for now. Changing
>> > mappings during DMA is stupid anyway: I don't think the guest can
>> > recover the results of DMA safely, even though it might be used on
>> > transfers in progress you simply don't care about anymore. Paul Brook
>> > suggested we could update the cpu_physical_memory_map() mappings
>> > somehow, but I think that's kinda difficult to accomplish.
>>
>> A malicious or broken guest shouldn't be able to crash or corrupt QEMU
>> process memory.  The IOMMU can only map from bus addresses to guest
>> physical RAM (?) so the worst the guest can do here is corrupt itself?
>>
> That's true, but it's fair to be concerned about the guest itself.
> Imagine it runs some possibly malicious apps which program the hardware
> to do DMA. That should be safe when a IOMMU is present.
>
> But suddenly the guest OS changes mappings and expects the IOMMU to
> enforce them as soon as invalidation commands are completed. The guest
> then reclaims the old space for other uses. This leaves an opportunity
> for those processes to corrupt or read sensitive data.

As long as QEMU acts in the same way as real hardware we should be
okay.  Will real hardware change the mappings immediately and abort
the DMA from the device if it tries to access an invalidated address?

Stefan

Re: Status update

[Eduard - Gabriel Munteanu]

On Fri, Jul 02, 2010 at 09:03:39AM +0100, Stefan Hajnoczi wrote:
> > That's true, but it's fair to be concerned about the guest itself.
> > Imagine it runs some possibly malicious apps which program the hardware
> > to do DMA. That should be safe when a IOMMU is present.
> >
> > But suddenly the guest OS changes mappings and expects the IOMMU to
> > enforce them as soon as invalidation commands are completed. The guest
> > then reclaims the old space for other uses. This leaves an opportunity
> > for those processes to corrupt or read sensitive data.
> 
> As long as QEMU acts in the same way as real hardware we should be
> okay.  Will real hardware change the mappings immediately and abort
> the DMA from the device if it tries to access an invalidated address?
> 
> Stefan

Real, non-IOMMU aware hardware looks like this (simplified):

Device <-> IOMMU <-> Memory	(1)

Most QEMU translated devices would do exactly that. But AIO is something
like this:

Device + <---translation--- IOMMU	(2)
       ^
       |
       ---------R/W--------> Memory

There are two reasons this form isn't reducible to the former in case of
AIO:
1. It uses cpu_physical_memory_map().
2. The actual I/O happens on a separate thread.

Real hardware of form (1) has no problems since all access is serialized
through the IOMMU. That doesn't mean mapping updates happen instantly.
But software can wait (and be notified) for the updates to happen.

Real hardware of form (2) is comprised of devices which have their own
IOTLB, I think. But unlike cpu_physical_memory_map() in software-land,
these mappings can be updated during I/O. (These devices are
necessarily IOMMU-aware.)

As I see it, this mmaping trick is quite crucial to AIO and I'm not sure
there's a way around it. The solution I proposed is making the IOMMU
wait* for AIO to finish. Perhaps we can also break mmaping into smaller
chunks so they complete in a reasonable time.

* - Waiting here means to delay notifying the guest OS that invalidation
commands completed. This is the important part: if the guest is told the
mappings have been updated, then they really have to be.


	Eduard

Status Update

[Richard Purdie]

As people know, feature freeze was at the start of the week. I've been
trying to ensure the remaining patches get rounded up and into the
build.

At the same time, the Yocto Project Autobuilder infrastructure has been
undergoing an upgrade to new software. I really wanted this to happen
now as if it doesn't it has to be postponed until after release and we
need features and efficiency in the new codebase.

Unfortunately its highlighted a number of issues, particularly races
and my bandwidth has been sucked into helping trying to fix the sanity
test failures. I have fast tracked some patches into master in the
qemu-testlib code with that in mind since the autobuilder is pretty
critical to how we maintain stability of OE-Core.

This combined with an abnormally high meeting load has meant there are
some things I haven't been able to do. I am unfortunately totally max'd
out atm, people's patience is appreciated.

Cheers,

Richard

Status Update

[Richard Purdie]

I've promised to start sending out some kind of status updates. I have a
few things that are worth communicating and we have a kind of fresh
start with 1.5 so here goes. This one turned out lengthier than I'd
like, I'm not planning on writing an essay in future!

I'm also hoping this email will save me repeating myself in meetings.

Pending Patches
===============

Having been concentrating on 1.4 release issues, I have then been
concentrating on overhauling the bugs in the bugzilla for the 1.5
planning rather than reviewing/merging patches. I just merged a load of
different patches into master thanks to some work from Saul pulling them
together and testing them and am working on going through any stragglers
as time permits.

1.4 Released
============

1.4 ("dylan") is now released. There seems to be a trend for people to
start testing late in the cycle and this means we can't get various
fixes into the release if issues are found. With this in mind I'm
proposing we follow up with a point release (1.4.1) in roughly six
weeks.

Point Releases
==============

I'd actually like to see us have faster turnaround on point releases.
Right now, the biggest bottleneck is the QA process since QAing a point
release *and* weekly 1.5 dev work is problematic. The QA automation work
planned for 1.5 should help with this a lot. In the meantime we're going
to do what we can for faster point release turnarounds including
planning a 1.3.2 soon as well.

1.5 Planning
============

There have been a lot of ideas put into the bugzilla. I along with
various other people have looked through them and tried to prioritise
things with a view to figuring out what we can do in 1.5. The themes are
now on https://wiki.yoctoproject.org/wiki/Yocto_1.5_Features which
summarise the key development targets we could pick out from the
individual ideas.

Plans are starting to get locked in but its not too late to add
enhancement requests to the bugzilla. If there is anything anyone feels
very strongly should be scheduled differently, please let me/us know.
There are a number of things going into the future state as we simply
don't have the resources to do everything, much as I'd like to.

Spending a little time "away"
=============================

The Yocto Project is very full on for me and I've been in the thick of
things since we started planning 0.9 back in 2010. For various reasons I
have a significant chunk of vacation time built up and to be honest, six
releases and three years later, I could use a break from things. There
never is, or will be "a good time" to do that.

I'm therefore planning to semi-disappear half way through the 1.5 cycle,
probably for around six weeks. The exact plan is still to be determined
but in that time I will ask a small team of people (Paul/Ross/Saul?) to
put pull requests together and I will handle those but I will step back
from a lot of the other day to day things.

Just to confuse things, in some of that time I will likely be playing
with some embedded devices and home automation so I may post the odd
patch but it will be as a user and I will not be doing my usual patch
review/architect role other than on the consolidated tree.

I'm warning people about this now so there are no excuses when people
need me for something and I'm not around. If you need input from me, get
it early in the 1.5 cycle.

Cheers,

Richard

Status Update

[Richard Purdie]

In summary its been a fairly quiet week, there are number of different
holidays in different geos.

Pending Patches
===============

There has been a steady stream of 1.5 patches starting to appear. With
Saul's help these have been tested on the autobuilder and then many have
been merged into master. There were some race issues with one of the
sstate performance improvements but that is now resolved as far as I
know. We're making slow/steady progress on gcc 4.8.

1.5 planning is well underway now and some people are starting to work
on the implementation of features.

1.4.1
=====

Paul is going to maintain the 1.4 series. He's already queued up several
of the fixes that didn't make 1.4 and these are now undergoing testing
on the autobuilder. If anyone has any changes to propose for 1.4.1, now
is the time. We're not looking at features or upgrades but will consider
bug fixes.

1.3.2
=====

I've merged a lot of queued changes that Ross prepared for 1.3.2 and
this is now undergoing testing.

Python 3
========

I've been asked when bitbake will work under python3 several times now.
The first question is which python 3 as there are big differences
between them. I'm still not sure which python 3.X we'd end up wanting to
target.

I've done a bit of research and the changes we need to make are mostly
of the annoying kind. I have had bitbake build a few things (e.g.
pseudo-native) with a hacked up bitbake and metadata. The current
roadblock is the oe.types functionality and I think I'll need Chris'
help to fix this since some functionality it relies on was removed in
3.x.

I've a series I'll send out which deals with the simple issues. 

There are several cases we'll have to put in "hacky" code to make things
work for both py2 and py3, the most annoying being octal file
permissions. There is no syntax which works in both so we're facing
having to write them in hexidecimal if we want both to work. When I have
spare time I'll continue to work on it and try and get a more complete
build working.

Cheers,

Richard

Re: Status Update

[Trevor Woerner]

On Tue, May 7, 2013 at 8:45 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> 1.3.2
> =====
>
> I've merged a lot of queued changes that Ross prepared for 1.3.2 and
> this is now undergoing testing.

I have a patch for Danny to allow qemu-native of qemu-1.2.0 to compile
on openSuSE 12.3 (which, apparently, is also affected by the same (or
a similar) DSO linking change that affects fedora). Are the queued
changes available somewhere so I can check this hasn't already been
applied?

Re: Status Update

[Burton, Ross]

On 7 May 2013 17:29, Trevor Woerner <twoerner@gmail.com> wrote:
>> 1.3.2
>> =====
>>
>> I've merged a lot of queued changes that Ross prepared for 1.3.2 and
>> this is now undergoing testing.
>
> I have a patch for Danny to allow qemu-native of qemu-1.2.0 to compile
> on openSuSE 12.3 (which, apparently, is also affected by the same (or
> a similar) DSO linking change that affects fedora). Are the queued
> changes available somewhere so I can check this hasn't already been
> applied?

The merged changes are in the "danny" branch of the oe-core and poky,
and there's a ross/danny-next branch in openembedded-core-contrib that
is my staging area for submissions to danny.

Ross

Re: Status Update

[Trevor Woerner]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

On Tue, May 7, 2013 at 1:13 PM, Burton, Ross <ross.burton@intel.com> wrote:
>> I have a patch for Danny to allow qemu-native of qemu-1.2.0 to compile
>> on openSuSE 12.3 (which, apparently, is also affected by the same (or
>> a similar) DSO linking change that affects fedora). Are the queued
>> changes available somewhere so I can check this hasn't already been
>> applied?
>
> The merged changes are in the "danny" branch of the oe-core and poky,
> and there's a ross/danny-next branch in openembedded-core-contrib that
> is my staging area for submissions to danny.

I've looked through the list of patches and don't seem to find
anything that will do what my patch needs to do in order to build
qemu-1.2.0 on openSuSE 12.3. Would you consider adding the attached
patch for Danny? (normally I'd send it in-line but I'm behind a
firewall at the moment that won't allow "git send-email" to succeed,
otherwise I could submit it this evening)

[-- Attachment #2: 0001-qemu-native-fix-DSO-linking.patch --]
[-- Type: application/octet-stream, Size: 2593 bytes --]

From a4ada65e68cece477ef760995ae3395cc7dd27f8 Mon Sep 17 00:00:00 2001
From: Trevor Woerner <twoerner@gmail.com>
Date: Tue, 7 May 2013 13:37:14 -0400
Subject: [PATCH][danny] qemu-native: fix DSO linking

I noticed this issue trying to build using the Danny branch on what is
currently the most recent openSuSE (12.3). It appears to be similar to the
fedora DSO linking issue:
https://fedoraproject.org/wiki/UnderstandingDSOLinkChange.

Upstream-status: Inappropriate [OE Specific]
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 .../qemu/qemu-1.2.0/linker-flags-2.patch           |   26 ++++++++++++++++++++
 meta/recipes-devtools/qemu/qemu_1.2.0.bb           |    1 +
 2 files changed, 27 insertions(+)
 create mode 100644 meta/recipes-devtools/qemu/qemu-1.2.0/linker-flags-2.patch

diff --git a/meta/recipes-devtools/qemu/qemu-1.2.0/linker-flags-2.patch b/meta/recipes-devtools/qemu/qemu-1.2.0/linker-flags-2.patch
new file mode 100644
index 0000000..ba50c8d
--- /dev/null
+++ b/meta/recipes-devtools/qemu/qemu-1.2.0/linker-flags-2.patch
@@ -0,0 +1,26 @@
+diff -urN qemu-1.2.0.orig/Makefile qemu-1.2.0/Makefile
+--- qemu-1.2.0.orig/Makefile	2012-09-05 10:03:06.000000000 -0400
++++ qemu-1.2.0/Makefile	2013-05-07 12:18:23.638560591 -0400
+@@ -172,7 +172,7 @@
+ qemu-img-cmds.h: $(SRC_PATH)/qemu-img-cmds.hx
+ 	$(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -h < $< > $@,"  GEN   $@")
+ 
+-qemu-ga$(EXESUF): LIBS = $(LIBS_QGA)
++qemu-ga$(EXESUF): LIBS = $(LIBS_QGA) -lrt
+ qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated
+ 
+ gen-out-type = $(subst .,-,$(suffix $@))
+diff -urN qemu-1.2.0.orig/configure qemu-1.2.0/configure
+--- qemu-1.2.0.orig/configure	2012-09-05 10:03:06.000000000 -0400
++++ qemu-1.2.0/configure	2013-05-07 12:11:06.940307455 -0400
+@@ -2681,6 +2681,10 @@
+   LIBS="-lrt $LIBS"
+ fi
+ 
++if test "$linux" = "yes" ; then
++  LIBS="-lrt $LIBS"
++fi
++
+ if test "$darwin" != "yes" -a "$mingw32" != "yes" -a "$solaris" != yes -a \
+         "$aix" != "yes" -a "$haiku" != "yes" ; then
+     libs_softmmu="-lutil $libs_softmmu"
diff --git a/meta/recipes-devtools/qemu/qemu_1.2.0.bb b/meta/recipes-devtools/qemu/qemu_1.2.0.bb
index 517a74b..7a81f0a 100644
--- a/meta/recipes-devtools/qemu/qemu_1.2.0.bb
+++ b/meta/recipes-devtools/qemu/qemu_1.2.0.bb
@@ -8,6 +8,7 @@ SRC_URI = "\
     file://powerpc_rom.bin \
     file://no-strip.patch \
     file://linker-flags.patch \
+    file://linker-flags-2.patch \
     file://qemu-vmware-vga-depth.patch \
     file://fix-configure-checks.patch \
     file://fallback-to-safe-mmap_min_addr.patch \
-- 
1.7.10.GIT

Re: Status Update

[Burton, Ross]

On 7 May 2013 18:57, Trevor Woerner <twoerner@gmail.com> wrote:
> I've looked through the list of patches and don't seem to find
> anything that will do what my patch needs to do in order to build
> qemu-1.2.0 on openSuSE 12.3. Would you consider adding the attached
> patch for Danny? (normally I'd send it in-line but I'm behind a
> firewall at the moment that won't allow "git send-email" to succeed,
> otherwise I could submit it this evening)

Merged to my danny-next branch, after confirming that this isn't
needed in Dylan/master as qemu 1.4 has the -lrt linkages declared.

Ross

Re: Status Update

[Trevor Woerner]

On Wed, May 8, 2013 at 6:35 AM, Burton, Ross <ross.burton@intel.com> wrote:
> On 7 May 2013 18:57, Trevor Woerner <twoerner@gmail.com> wrote:
>> I've looked through the list of patches and don't seem to find
>> anything that will do what my patch needs to do in order to build
>> qemu-1.2.0 on openSuSE 12.3. Would you consider adding the attached
>> patch for Danny? (normally I'd send it in-line but I'm behind a
>> firewall at the moment that won't allow "git send-email" to succeed,
>> otherwise I could submit it this evening)
>
> Merged to my danny-next branch, after confirming that this isn't
> needed in Dylan/master as qemu 1.4 has the -lrt linkages declared.

Whoops, sorry. I guess I should have mentioned that (since I was aware
that qemu-native works fine with the latest Yocto HEAD).

Status Update

[Richard Purdie]

Its been a few weeks since I sent one of these out, mainly as I've not
had much to report.

Pending Patches
===============

Patches have been steadily getting reviewed and merged. Some things took
a little longer than is ideal due to travel on Saul and my part. I know
Saul is still travelling.

Autobuilder status is much improved compared to where its been the past
couple of weeks. We added a number of new sanity tests, they found
issues, we've now got many of them fixed. The nightly-qa test is still
problematic, the fsl-ppc machines are continuing to fail and have been
failing for a while now :(.

Automated QA
============

Architecture/planning wise, there is a big shakeup of the automated QA
tests coming down the pipeline for 1.5. The current shell script based
system isn't scaling so we're looking at moving to python unittest.

Why is this important? Manual QA currently uses a lot of or QA resources
and means releases take longer than we'd like, we also have issues if we
ever try two in parallel (e.g. a point release). I want to see this
problem removed and automation looks like the best way to achieve it. It
means the QA teams can then focus on more automation and writing new
test cases for new features and so on.

Once the initial conversion happens, we're looking to convert many of
the currently manual tests over to become automated and hence improve
the speed we can put releases through the QA process amongst other
things.

This is all still being planned, further discussion will no doubt happen
as soon as there is a plan to propose/discuss. There are a variety of
open bugs in the bugzilla and people interested in what is going on can
look at those and see the rough plans for 1.5.

I can quickly summarise that:

a) this work is focused on testing with qemu, not target hardware.
Nothing done in this work would be incompatible with real hardware, its
just not a focus right now (maybe 1.6?)

b) We plan to leverage existing tests where we can, e.g. "make check" on
target using ptest packages

c) initially the plan is to cover the runtime qemu tests

d) we're also looking at some kind of oe-selftest, a bit like
bitbake-selftest which would run "bitbake X, change Y, bitbake X, check
something" type scenarios.

e) extensions to the package QA tests (insane.bbclass) are being
considered

f) a new level of QA tests against rootfs after they're generated will
likely be created

The plan is to establish the new python unittest code using a small
team, then open things up for more contributions of test cases since
this part of the process can be parallelised.

1.3.2
=====

It shipped! Thanks Ross! :)

Hopefully this should round out the bulk of the 1.3 issues people were
running into. There is no 1.3.3 planned at this point.

1.4.1
=====

A number of fixes were merged for this. I know there are a small number
of patches still to come in but we'll need to start putting this through
the release process shortly so please get any requests for bugs fixes
for 1.4.x to Paul ASAP.

Bitbake
=======

I've been spending quite a bit of time reviewing and also writing some
changes to the core of bitbake, splitting up cooker. These changes set
the scene for webhob and the memory resident bitbake concept as well as
removal of the double bitbake execution. There will be more changes in
this area landing and there may be some further mild instability whilst
we work though any issues.

There have also been some patches which remove the "global method scope"
we had in bitbake. Those patches merged, I've not heard of anyone with
issues due to them yet (the parallel parsing already half broke the
global scope anyway).

Cheers,

Richard

Re: Status Update

[Jack Mitchell]

On 28/05/2013 18:35, Richard Purdie wrote:
>
> <snip>
>

Richard,

These updates are really informative, thank you for taking the time to 
pull together and summarize.

Cheers,

-- 

   Jack Mitchell (jack@embed.me.uk)
   Embedded Systems Engineer
   http://www.embed.me.uk

-- 

Status Update

[Richard Purdie]

As previously mentioned, I'm planning on taking some vacation time which
starts today and will be until mid August. I will still be around a
little bit but not actively reviewing patches and so on as I'd usually
be doing.

In my absence, Paul, Ross and Saul have kindly agreed to review patches
and queue them up, I will stick my head in occasionally and hopefully
just need to merge the queue. 

Cheers,

Richard

Status Update

[Richard Purdie]

Its been a while since I sent one of these out, I never got back into
the habit after I was away last year. I'm told they're useful to various
people and they stop me having to repeat myself to people so I'm going
to give them another go.

Pending Patches
===============

The feature freeze point for the 1.6 release is today. Some things of
note that have merged recently including:

* Reworked ext2/3/4 rootfs generation. Great work to the people involved
  in getting the patches upstream! Can we delete genext2fs now? Please?
* ToasterUI Updates
* Systemd Upgrade to 209/210ish
* lttng updates
* core-image-basic renaming
* PRINC deprecation warning
* autotools aclocal improvements
* using the separate build include by default
* bitbake manual updates
* various oe-selftest improvements/tweaks

Its good to see many of these coming in, some have been long awaited.

I'm aware we don't quite have some things yet, in particular the 3.14
kernel so there are some changes to still to come in. We've been testing
the -dev kernel on the autobuilder, we have a list of issues we need to
resolve (perf mainly at this point) and as soon as 3.14 is officially
released we'll be good to go. Other things I'm aware of as pending for
1.6 are:

* gummiboot recipes (various versions exist in various layers, this 
  pulls things together hopefully)
* oe-init-build-env tweaks from Gary
* some parts of the "on hardware" automated testing
* possibly some bitbake fetcher extension code for automated "upgrade" 
  detection
* Enabling Separate B from S by default. Martin asked me to hold off 
  this until the issues form the previous changes get sorted out. 
  OE-Core is ready for it, the question is the other layers. We 
  probably need to bump the tmpdir ABI number for this change.

I get the feeling there may be some other things which I don't have on
the list too, please let me know of major pending features I don't have
here.

There are also some things looking very unlikely to get done in 1.6 at
this point. These include:

* Memory Resident and Interactive Bitbake work
* Kernel installation footprint optimisation
* python devshell
* PR Server improvements
* Locked SState support

Patches for some parts of these exist but we're running out of runway to
get them into a state ready to merge.

The recent autobuilder changes did cause a backlog of patches waiting
for testing. On the positive side the autobuilder does seem to have
stabilised now and the changes present there have been worth waiting
for. I believe we have caught up with most of the backlog now and had a
mostly green build on Friday.


Bitbake Changes
===============

We found there were some issues when SIGTERM was sent to bitbake's
various (sub)processes, as might happen on a buildbot or jenkins setup
when stopping builds. The exact effect depended on the order the
processes received the signal and ranged from locking up to using 100%
CPU. There is a fairly comprehensive set of patches on the bitbake-devel
list which should make things behave better.

I've also been a little frustrated with the latency of certain bitbake
commands, it seems to do nothing at some points and debugging confirmed
that (e.g. setting the server/process.py select call timeout to 5s made
the issues extremely obvious). Again, there are patches out to try and
optimise out the various delays which are pointless and bitbake feels
snappier as a result, at least to me anyway.

I'd also like to give a mention to the BitBake manual once again. Its
been heavily reworked, improved and brought up to date. The edits are
now in bitbake master branch.

1.4.3
=====

This was held back due to the recent gnutls CVE, its now undergoing
testing and should be ready for release soon if that is successful. If
not, it will have to wait until after 1.6 ships.


Cheers,

Richard

Re: Status Update

[Robert Yang]

On 03/10/2014 10:20 AM, Richard Purdie wrote:
> Its been a while since I sent one of these out, I never got back into
> the habit after I was away last year. I'm told they're useful to various
> people and they stop me having to repeat myself to people so I'm going
> to give them another go.
>
> Pending Patches
> ===============
>
> The feature freeze point for the 1.6 release is today. Some things of
> note that have merged recently including:
>
> * Reworked ext2/3/4 rootfs generation. Great work to the people involved
>    in getting the patches upstream! Can we delete genext2fs now? Please?

Yes, I think that we can delete the genext2fs since the "mke2fs -d" can
do the same thing as genext2fs.

// Robert

> * ToasterUI Updates
> * Systemd Upgrade to 209/210ish
> * lttng updates
> * core-image-basic renaming
> * PRINC deprecation warning
> * autotools aclocal improvements
> * using the separate build include by default
> * bitbake manual updates
> * various oe-selftest improvements/tweaks
>
> Its good to see many of these coming in, some have been long awaited.
>
> I'm aware we don't quite have some things yet, in particular the 3.14
> kernel so there are some changes to still to come in. We've been testing
> the -dev kernel on the autobuilder, we have a list of issues we need to
> resolve (perf mainly at this point) and as soon as 3.14 is officially
> released we'll be good to go. Other things I'm aware of as pending for
> 1.6 are:
>
> * gummiboot recipes (various versions exist in various layers, this
>    pulls things together hopefully)
> * oe-init-build-env tweaks from Gary
> * some parts of the "on hardware" automated testing
> * possibly some bitbake fetcher extension code for automated "upgrade"
>    detection
> * Enabling Separate B from S by default. Martin asked me to hold off
>    this until the issues form the previous changes get sorted out.
>    OE-Core is ready for it, the question is the other layers. We
>    probably need to bump the tmpdir ABI number for this change.
>
> I get the feeling there may be some other things which I don't have on
> the list too, please let me know of major pending features I don't have
> here.
>
> There are also some things looking very unlikely to get done in 1.6 at
> this point. These include:
>
> * Memory Resident and Interactive Bitbake work
> * Kernel installation footprint optimisation
> * python devshell
> * PR Server improvements
> * Locked SState support
>
> Patches for some parts of these exist but we're running out of runway to
> get them into a state ready to merge.
>
> The recent autobuilder changes did cause a backlog of patches waiting
> for testing. On the positive side the autobuilder does seem to have
> stabilised now and the changes present there have been worth waiting
> for. I believe we have caught up with most of the backlog now and had a
> mostly green build on Friday.
>
>
> Bitbake Changes
> ===============
>
> We found there were some issues when SIGTERM was sent to bitbake's
> various (sub)processes, as might happen on a buildbot or jenkins setup
> when stopping builds. The exact effect depended on the order the
> processes received the signal and ranged from locking up to using 100%
> CPU. There is a fairly comprehensive set of patches on the bitbake-devel
> list which should make things behave better.
>
> I've also been a little frustrated with the latency of certain bitbake
> commands, it seems to do nothing at some points and debugging confirmed
> that (e.g. setting the server/process.py select call timeout to 5s made
> the issues extremely obvious). Again, there are patches out to try and
> optimise out the various delays which are pointless and bitbake feels
> snappier as a result, at least to me anyway.
>
> I'd also like to give a mention to the BitBake manual once again. Its
> been heavily reworked, improved and brought up to date. The edits are
> now in bitbake master branch.
>
> 1.4.3
> =====
>
> This was held back due to the recent gnutls CVE, its now undergoing
> testing and should be ready for release soon if that is successful. If
> not, it will have to wait until after 1.6 ships.
>
>
> Cheers,
>
> Richard
>
>

Status Update

[Richard Purdie]

I was travelling in the last week which limited the time I could spend
on various issues.

The week has mostly been seeing patches being tested on the autobuilder
in MUT/master-next with a view to ironing out bugs before code merges.

Pending Patches
===============

I did manage to flush some queued patches into the various repositories
after testing on the autobuilder. The current pending list is now:

* archiver rework patches (not had a chance to review yet)
* 3.14 kernel (not released yet, we have patches being tested, ongoing 
  perf issues but close to resolution with any luck)
* updated meta-yocto-bsp bsps for BBB and edge router
* Enabling B != S - Can't decide whether to push this or not. Have got 
  several known issues fixed up in layers (thanks Otavio). Martin, any 
  thoughts on this?
* pusleaudio 5.0?
* systemd 112?
* bitbake fetcher extension code for automated "upgrade" detection

We have had a number of these through testing cycles on the autobuilder,
systemd in particular should be ready to go in and I'll probably merge
that.

I'm not happy about the changes coming in as late as they are, equally,
I think the changes are work getting in. It will be ready when it is
ready...

Bitbake Changes
===============

The sigchld handler is continue to cause issues. Some are fixed but we
were seeing locked up processes on the autobuilder which appeared to be
trapped in waitpid. I've pushed some further patches to stop the code
doing bad things, I'm not 100% sure why it was breaking but the latest
builds seem to be behaving.

I'd advice people to keep up to date with master and report any issues
they find. Bonus marks for good reproducers, it makes things much faster
to fix!

I do want to get -S fixed for the release so some changes are pending
for that.

1.4.3
=====

There were sstate issues on the autobuilder, Paul backported the patch
which was confirmed as fixing the issue. We tried for another build, I
messed up the branches though (sorry Paul/Saul/Beth). We'll spin another
build when the autobuilder is free.


Cheers,

Richard

Status Update

[Richard Purdie]

Executive Summary
=================

A very rocky week but we now have consistent green builds on the
autobuilder.

1.6 Release Status
===============

Over the last week things have stabilised a lot from my perspective.
Most of the major changes are now in or have at least been tested. In
particular the linux-yocto-dev kernel (3.14) is now cleanly building so
whilst the change is coming in late, it comparatively well tested.

The reference BSPs are also set to change and patches are being tested
for those. It is late in the cycle for them however this was we get to
close out a long standing set of bugs that are unlikely to get fixed and
replace them with a new set of things which people are much more likely
to work on.

The archiver changes are in, there is a second set of patches pending.
I'd appreciate eyes on this from people who use the code.

I'm not going to push the B != S change in this release. It will land
immediately into master as soon as we branch for 1.6 though.

systemd and pulseaudio merged in the end, not least since the systemd
changes appear to address some of the sanity failures we've been seeing.

Autobuilder
===========

We've had a rough ride over the last week. We kept managing to "break"
slaves, one with disk corruption problems (RAID firmware?), another with
power supply issues and a variety of other issues. This took us from
seven to four slaves.

We also had a build fail when fork() stopped working, probably due to
resource issues. As such we've dropped from 3 to 2 concurrent builds on
the builders. There were also various software issues with disks not
being cleared properly and some autobuilder git fetcher code problems.

Thanks to hard work from Beth and Michael, we're now back at autobuilder
full slave count and we have updated autobuilder code which showed great
promise over the weekend. We now have green builds for 1.6 with two
exceptions, the uclibc world build and qa-targetbuilds targets, both
only recently added to the system.

Bitbake Changes
===============

The sig child handler was a disaster, I ended up reverting the code to
use subprocess' poll() method. The handler would be more efficient but
doesn't mix with subprocess well in python 2.7.3. We were seeing hanging
bitbake processes, we should now have all those issues resolved for
release.

Cheers,

Richard

Status Update

[Richard Purdie]

Executive Summary
=================

3.14 kernel is in and we have green builds. Managed to make a dent in
the defect tracking metrics but work remains. Trending well for the
release, need to keep up the bug fixing.

1.6 Release Status
===============

The 3.14 kernel was released, we were able to updated the patches we had
queued ready and it merged fairly smoothly. The builds last night after
3.14 was integrated were all green which is nice to see. This included
the switch to the two new reference BSPs. This was a big risk factor for
the release so its nice to see it happen smoothly, finally.

I've started aggressively trying to sort out my bug backlog, I've
managed to close about 10 of them over the past week, some were found
not to be bugs when they were delved into, some were resolved. There are
some interesting bugs to do with sstate reuse too. My remaining 6 bugs
(ignoring enhancements) are probably 1.7 material at this point. I know
others have been making an effort too, including in other areas like
documentation. We still have some runway before release so lets use that
time to best advantage and see if we can lower the bug tracking metrics
further.

I have been trying to merge bug fixes quickly. I have to say that there
has been a sharp drop in quality of patches on the list over the last
week though. As people rush to try and get things in, its clear less
testing is happening and this puts the build at risk. I'm therefore
starting to be quite strict on review and if something is rushed and
buggy, it (and any subsequent versions) will move to the bottom of the
priority list.

Autobuilder
===========

This seems to be stabilising. There has been the odd setup issue,
another AB had RAID issues and the odd regression crept into the
autobuilder code but things are being dealt with and feel under control.

Bitbake Changes
===============

There is still some churn on the toaster codebase but given its new
code, that is probably only to be expected. There have been some useful
sstate signature fixes merged. I'm particularly interested in reports
from people where sstate should have been reused and it wasn't. I'd
really like to get to the bottom of these issues once and for all.

Cheers,

Richard

Status Update

[Richard Purdie]

Executive Summary
=================

Continuing to trend well for release, bug fixes continue to be merged
although patches are starting to get deferred for 1.7 if they're too
risky at this point.


1.6 Release Status
===============

The summary says it all really, bug trends are good but work remains.
The openssl CVE needs to get addresses so we'll probably take a package
upgrade for that specific component.

I am continuing to see a lot of "dubious" patches where the rationale
for the fix doesn't add up. I'm leaning not to take these in those
cases.


Autobuilder
===========

Builds are greenish however we've had a lot of problem with the sanity
tests suffering hung processes. There have also been buildslave issues,
particularly with ab05 which has now been taken out the pool pending
investigation as to why its "upset", so its hard to know if issues are
the hardware or the sanity tests.

Cheers,

Richard

Status Update

[Richard Purdie]

Changes in the last week
========================

The past week has been turbulent. It seemed we were on track early last
week, then we found some problems. The -rc3 build suffered some build
failures, particularly a sysroot issue with perf. There were also some
BSP issues with beaglebone as well as some QA challenges.

Were able to pull together all the needed fixes quickly and in the end
we built out an -rc4 early, dropping -rc3 since there were too many
known issues. We have now branched although I've not started merging 1.7
patches as yet and hope to hold off for a short while yet.

The biggest problem right now is that beagebone doesn't boot depending
on the directory depth the kernel is built within. We don't understand
why.

I had some issues with email this week. We think of it as instantaneous
but a load of mine disappeared onto the secondary MX and I didn't get it
until after -rc4 was built. This is why some patches didn't get dealt
with so my apologies for that, the timing was particularly unfortunate.

I don't believe we have a reason to stop 1.6 at this point, particularly
if we can document the beaglebone issue. The -rc4 QA report is due soon
and will drive the release decisions.

I've not had much time to spend on 1.7 planning at this point but I'm
hoping to get to that next.

Cheers,

Richard

Status Update

[Richard Purdie]

Executive Summary
=================

The -rc4 build of 1.6 was ready for release with some last minute fixes
for the beaglebone issues (thanks!). Its great to finally get that out
there.

I've started merging changes into master. I've put various gcc changes
forward that I've had a few moments to work on. There was also an issue
I found in bitbake which gives about a 7% build speed improvement on our
standard image test and I found some versions of git are rather slow at
operations which should be fast.

I've not travelled to ELC/OEDAM due to being unwell and worn out :(

1.6 Release
===========

I believe this is ready for release. There are a variety of patches
which did not make it in and we will queue those for a point release.
The timing on the point release will depend on the kind of issues we
find in 1.6.

Master
======

I've started merging patches in so this has opened for changes. Of note
so far:

GCC - I did find a few moments to write most of the patches needed to
rework our parts of our gcc recipes and continue to improve the
toolchain which will be valuable as we more forward. The first part of
these has merged. The second part is out there for review and there are
some bugs there I need to fix before it merges (meta-ide-support in
particular). There is a third set of patches I've not cleaned up and
sent out yet which standardise the toolchain hashes.

Bitbake - The task scheduling algorithm has a couple of bugs in it which
I found after noticing some strange behaviour with low numbers of
threads. This was worth a 7% speedup of our benchmark image build test.
Unfortunately it was too late to get that into 1.6 but it may make the
next point release. I also found that git 1.8 is slow for some
operations and we really want to use 1.9+ (worth around 1 minute on
linux-yocto kernel build time). Chris has also found what looks like a
nasty bug in the codeparser cache which is a good thing to find.

ELC/OEDAM
=========

I've not travelled to ELC/OEDAM as planned. I was already worn out with
previous travel and a variety of other things, then I've come down with
a bug which I held off over the weekend but has caught up with me
now :(. The flights and so on don't agree with me at the best of times
and I'd lost my voice before even leaving so I decided to rest instead.
It wouldn't have been fair to potentially spread the bug around either.

Its the first trip I've ever cancelled and I'm sad not to be at OEDAM in
particular but if I had travelled I doubt I would have been in any state
to participate. Sorry if that causes difficulties for anyone.

I'm aiming to write a separate email about where I think we need to
focus with the project going forward.

Given the situation, I'm going to try rest this week and take a bit of a
break from things. This may slow down patch merging and so on.


Cheers,

Richard

Re: Status Update

[Richard Purdie]

On Tue, 2014-04-29 at 09:49 +0100, Richard Purdie wrote:
> Master
> ======
> 
> I've started merging patches in so this has opened for changes. Of note
> so far:
> 
> GCC - I did find a few moments to write most of the patches needed to
> rework our parts of our gcc recipes and continue to improve the
> toolchain which will be valuable as we more forward. The first part of
> these has merged. The second part is out there for review and there are
> some bugs there I need to fix before it merges (meta-ide-support in
> particular). There is a third set of patches I've not cleaned up and
> sent out yet which standardise the toolchain hashes.
> 
> Bitbake - The task scheduling algorithm has a couple of bugs in it which
> I found after noticing some strange behaviour with low numbers of
> threads. This was worth a 7% speedup of our benchmark image build test.
> Unfortunately it was too late to get that into 1.6 but it may make the
> next point release. I also found that git 1.8 is slow for some
> operations and we really want to use 1.9+ (worth around 1 minute on
> linux-yocto kernel build time). Chris has also found what looks like a
> nasty bug in the codeparser cache which is a good thing to find.

B != S - I meant to add here that I merged the B != S patch for
autotools which does have impact for other layers. I did work a while
back to fix up oe-core and the layers the yocto autobuilder tests. We've
held off this for a while and the hope was other layers would get tested
and fixed up. I'm not sure that has happened, particularly in meta-oe as
yet unfortunately. The benefits to B != S are significant in build
accuracy and determinism and I did say I'd merge it after release so
that has now happened.

PRINC - I have dropped the removal of PRINC for this cycle since there
were concerns about that and its probably a little aggressive.

contains - I also have concerns about the contains() changes being a
touch too aggressive and needing a little more soak time. I had assumed
they just standardised references to the function, not removal of the
functions themselves yet. I'm therefore holding off some of the bitbake
side of those changes.

Cheers,

Richard

Re: Status Update

[Otavio Salvador]

On Tue, Apr 29, 2014 at 8:29 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2014-04-29 at 09:49 +0100, Richard Purdie wrote:
> contains - I also have concerns about the contains() changes being a
> touch too aggressive and needing a little more soak time. I had assumed
> they just standardised references to the function, not removal of the
> functions themselves yet. I'm therefore holding off some of the bitbake
> side of those changes.

Could you add contains_any patch in meanwhile? I understand you want
to keep it rest for some time now before dropping the code of
oe.utils.contains.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Status Update

[Richard Purdie]

As people know we're working towards the release and I thought an update
might be useful.

There was a mixup about dates and the rc1 build happened a week early.
The first real rc2 was this Tuesday. We merged a variety of fixes into
it and it did build out. Unfortunately there were some late kernel
patches that came in after it, we then also found the toolchains in the
ADT/SDK were basically DoA. We decided to skip QA on rc2 and move to rc3
immediately.

The rc3 build did throw bogus QA warnings on the build due to dmesg
'errors' after the kernel change. We've also found that whilst the SDK
is ok, ADT is still broken :(. We are putting that one through QA
though.

We've also found that builds on xfs with 64 bit inodes are corrupting
permissions in pseudo (e.g. /usr/bin/python loses its execute bits). We
have a fix for that in pseudo master and have confirmed it fixes things
on the xfs build machine. This may explain some of the pseudo
permissions issues people have intermittently reported.

So we will run an rc4 build early next week and we're looking only to
take a small number of fixes into it, hopefully that one can become the
release and we can be done before ELC-E.

Cheers,

Richard