[Build App RFC 0/5] Review Request for Build Appliance

[Build App RFC 0/5] Review Request for Build Appliance

[Saul Wold]

Richard,

This is an updated set of changes for the Build Appliance to use
directdisk.  I am submitting this even as I am finding out that
there is an issue with genext2fs building Large FS Size (5GB and
greater).

The other option is to add a newer version of dosfstools, which is
not fully vetted yet, but may allow us to use bootimg instead of 
driectdisk.

Please review and comment the builder, image-vmdk and boot-directdisk
patches ready to go, the VMDK, update for directdisk is questionable
until we can solve the large image issues.

Sau!

The following changes since commit 647595821870693186c9ecad1bdfa12fe4ea334d:

  core-image-minimal-mtdutils.bb: Simple DESCRIPTION grammar fix. (2012-03-25 12:27:38 +0100)

are available in the git repository at:
  git://git.openembedded.org/openembedded-core-contrib sgw/self
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=sgw/oe-self

Dexuan Cui (1):
  self-hosted-image: pre-populate the builder user with poky source

Saul Wold (4):
  image-vmdk: Add symbolic link
  boot-directdisk: Fix Block Calcuation
  vmdk: Update for direstdisk
  builder: Enable auto starting of Hob

 meta/classes/boot-directdisk.bbclass               |   19 ++++++---
 meta/classes/image-vmdk.bbclass                    |   17 +++++++-
 meta/recipes-core/images/self-hosted-image.bb      |   41 +++++++++++++++++++-
 meta/recipes-graphics/builder/builder_0.1.bb       |    2 +-
 .../builder/files/builder_hob_start.sh             |   10 +++-
 5 files changed, 74 insertions(+), 15 deletions(-)

-- 
1.7.7.6

[Build App RFC 1/5] self-hosted-image: pre-populate the builder user with poky source

[Saul Wold]

From: Dexuan Cui <dexuan.cui@intel.com>

This patch installs the poky source into the /home/builder/poky/ of the
self-hosted-image.
This makes the user of self-hosted-image easier to start a build.

I think the recent poky master is stable enough, so I specify
a commit number by SRCREV -- we may want to update this number before
releasing 1.2.

This patch fixes [YOCTO #2065]

Signed-off-by: Dexuan Cui <dexuan.cui@intel.com>

Added code for supporting target based pseudo
fixed indentation

Signed-off-by: Saul Wold <sgw@linux.intel.com>
---
 meta/recipes-core/images/self-hosted-image.bb |   41 +++++++++++++++++++++++-
 1 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/images/self-hosted-image.bb b/meta/recipes-core/images/self-hosted-image.bb
index d56c2cb..5aa670d 100644
--- a/meta/recipes-core/images/self-hosted-image.bb
+++ b/meta/recipes-core/images/self-hosted-image.bb
@@ -4,7 +4,7 @@ LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3b58 \
                     file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 
-PR = "r5"
+PR = "r6"
 
 CORE_IMAGE_EXTRA_INSTALL = "\
     task-self-hosted \
@@ -13,7 +13,10 @@ CORE_IMAGE_EXTRA_INSTALL = "\
 IMAGE_FEATURES += "x11-mini package-management"
 
 # Ensure there's enough space to do a core-image-minimal build, with rm_work enabled
-IMAGE_ROOTFS_EXTRA_SPACE = "2621440"
+IMAGE_ROOTFS_EXTRA_SPACE = "1048576"
+#IMAGE_ROOTFS_EXTRA_SPACE = "2621440"
+#IMAGE_ROOTFS_EXTRA_SPACE = "20971520"
+#IMAGE_ROOTFS_EXTRA_SPACE = "5242880"
 
 # Do a quiet boot with limited console messages
 APPEND += "quiet"
@@ -21,3 +24,37 @@ APPEND += "quiet"
 IMAGE_FSTYPES = "vmdk"
 
 inherit core-image
+
+SRCREV = "26a46938d3ea1821e7bec4fa6cc8379babad238b"
+SRC_URI = "git://git.yoctoproject.org/poky;protocol=git"
+
+fakeroot do_populate_poky_src () {
+	# Because fetch2's git's unpack uses -s cloneflag, the unpacked git repo
+	# will become invalid in the target.
+	rm -rf ${WORKDIR}/git/.git
+	rm -f ${WORKDIR}/git/.gitignore
+
+	cp -Rp ${WORKDIR}/git ${IMAGE_ROOTFS}/home/builder/poky
+
+	mkdir -p ${IMAGE_ROOTFS}/home/builder/poky/build/conf
+	cp -Rp ${DL_DIR} ${IMAGE_ROOTFS}/home/builder/poky/build
+	echo "/usr/bin" > ${IMAGE_ROOTFS}/home/builder/poky/build/pseudodone
+	echo "BB_NO_NETWORK = \"1\"" > ${IMAGE_ROOTFS}/home/builder/poky/build/conf/auto.conf
+	echo "INHERIT += \"rm_work\"" >> ${IMAGE_ROOTFS}/home/builder/poky/build/conf/auto.conf
+        mkdir -p ${IMAGE_ROOTFS}/home/builder/pseudo
+        echo "export PSEUDO_PREFIX=/usr" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+        echo "export PSEUDO_LOCALSTATEDIR=/home/builder/pseudo" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+        echo "export PSEUDO_LIBDIR=/usr/lib/pseudo/lib64" >> ${IMAGE_ROOTFS}/home/builder/.bashrc
+
+        chown builder.builder ${IMAGE_ROOTFS}/home/builder/pseudo
+
+	chown -R builder.builder  ${IMAGE_ROOTFS}/home/builder/poky
+}
+
+IMAGE_PREPROCESS_COMMAND += "do_populate_poky_src; "
+
+python do_get_poky_src () {
+    bb.build.exec_func('base_do_fetch', d)
+    bb.build.exec_func('base_do_unpack', d)
+}
+addtask do_get_poky_src before do_rootfs
-- 
1.7.7.6

[Build App RFC 2/5] image-vmdk: Add symbolic link

[Saul Wold]

Signed-off-by: Saul Wold <sgw@linux.intel.com>
---
 meta/classes/image-vmdk.bbclass |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/meta/classes/image-vmdk.bbclass b/meta/classes/image-vmdk.bbclass
index 3e63006..9d596cd 100644
--- a/meta/classes/image-vmdk.bbclass
+++ b/meta/classes/image-vmdk.bbclass
@@ -10,6 +10,8 @@ inherit image-live
 
 create_vmdk_image () {
 	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
+	ln -s ${IMAGE_NAME}.vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.vmdk
+
 }
 
 python do_vmdkimg() {
-- 
1.7.7.6

Re: [Build App RFC 2/5] image-vmdk: Add symbolic link

[Darren Hart]

The idea being what? To keep a short-name symlink to the longer named
actual image?

Please include a commit message.

On 03/25/2012 11:47 PM, Saul Wold wrote:
> Signed-off-by: Saul Wold <sgw@linux.intel.com>
> ---
>  meta/classes/image-vmdk.bbclass |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/meta/classes/image-vmdk.bbclass b/meta/classes/image-vmdk.bbclass
> index 3e63006..9d596cd 100644
> --- a/meta/classes/image-vmdk.bbclass
> +++ b/meta/classes/image-vmdk.bbclass
> @@ -10,6 +10,8 @@ inherit image-live
>  
>  create_vmdk_image () {
>  	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
> +	ln -s ${IMAGE_NAME}.vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.vmdk
> +
>  }
>  
>  python do_vmdkimg() {

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

[Build App RFC 3/5] boot-directdisk: Fix Block Calcuation

[Saul Wold]

This also changes the timeout to be settable

The block calcuation was not correctly rounding, see comment

Thanks to Darren Hart for fixing this.

Cc: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
---
 meta/classes/boot-directdisk.bbclass |   19 +++++++++++++------
 1 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/meta/classes/boot-directdisk.bbclass b/meta/classes/boot-directdisk.bbclass
index 893164f..e9e1ba3 100644
--- a/meta/classes/boot-directdisk.bbclass
+++ b/meta/classes/boot-directdisk.bbclass
@@ -37,9 +37,8 @@ BOOTDD_EXTRA_SPACE ?= "16384"
 # Get the build_syslinux_cfg() function from the syslinux class
 
 AUTO_SYSLINUXCFG = "1"
-LABELS = "boot"
 SYSLINUX_ROOT ?= "root=/dev/sda2"
-SYSLINUX_TIMEOUT = "10" # 1 second
+SYSLINUX_TIMEOUT ?= "10"
 
 SYSLINUXCFG  = "${HDDDIR}/syslinux.cfg"
 SYSLINUXMENU = "${HDDDIR}/menu"
@@ -55,15 +54,23 @@ build_boot_dd() {
 	install -m 444 ${STAGING_LIBDIR}/syslinux/ldlinux.sys ${HDDDIR}/ldlinux.sys
 
 	BLOCKS=`du -bks ${HDDDIR} | cut -f 1`
-	SIZE=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
+	BLOCKS=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
+
+	# Ensure total sectors is an integral number of sectors per
+	# track or mcopy will complain. Sectors are 512 bytes, and we
+	# generate images with 32 sectors per track. This calculation is
+	# done in blocks, thus the mod by 16 instead of 32.
+	BLOCKS=$(expr $BLOCKS + $(expr 16 - $(expr $BLOCKS % 16)))
+
+	mkdosfs -n ${BOOTDD_VOLUME_ID} -S 512 -C ${HDDIMG} $BLOCKS 
+	mcopy -i ${HDDIMG} -s ${HDDDIR}/* ::/
 
-	mkdosfs -n ${BOOTDD_VOLUME_ID} -d ${HDDDIR} -C ${HDDIMG} $SIZE 
 	syslinux ${HDDIMG}
 	chmod 644 ${HDDIMG}
 
 	ROOTFSBLOCKS=`du -Lbks ${ROOTFS} | cut -f 1`
-	TOTALSIZE=`expr $SIZE + $ROOTFSBLOCKS`
-	END1=`expr $SIZE \* 1024`
+	TOTALSIZE=`expr $BLOCKS + $ROOTFSBLOCKS`
+	END1=`expr $BLOCKS \* 1024`
 	END2=`expr $END1 + 512`
 	END3=`expr \( $ROOTFSBLOCKS \* 1024 \) + $END1`
 
-- 
1.7.7.6

Re: [Build App RFC 3/5] boot-directdisk: Fix Block Calcuation

[Darren Hart]

On 03/25/2012 11:47 PM, Saul Wold wrote:
> This also changes the timeout to be settable
> 
> The block calcuation was not correctly rounding, see comment
> 
> Thanks to Darren Hart for fixing this.
> 
> Cc: Darren Hart <dvhart@linux.intel.com>

Thanks, but be sure to use the option to get that added to actual email
header :-)  That's "-c" for the send-pull-request script.

--
Darren

> Signed-off-by: Saul Wold <sgw@linux.intel.com>
> ---
>  meta/classes/boot-directdisk.bbclass |   19 +++++++++++++------
>  1 files changed, 13 insertions(+), 6 deletions(-)
> 
> diff --git a/meta/classes/boot-directdisk.bbclass b/meta/classes/boot-directdisk.bbclass
> index 893164f..e9e1ba3 100644
> --- a/meta/classes/boot-directdisk.bbclass
> +++ b/meta/classes/boot-directdisk.bbclass
> @@ -37,9 +37,8 @@ BOOTDD_EXTRA_SPACE ?= "16384"
>  # Get the build_syslinux_cfg() function from the syslinux class
>  
>  AUTO_SYSLINUXCFG = "1"
> -LABELS = "boot"
>  SYSLINUX_ROOT ?= "root=/dev/sda2"
> -SYSLINUX_TIMEOUT = "10" # 1 second
> +SYSLINUX_TIMEOUT ?= "10"
>  
>  SYSLINUXCFG  = "${HDDDIR}/syslinux.cfg"
>  SYSLINUXMENU = "${HDDDIR}/menu"
> @@ -55,15 +54,23 @@ build_boot_dd() {
>  	install -m 444 ${STAGING_LIBDIR}/syslinux/ldlinux.sys ${HDDDIR}/ldlinux.sys
>  
>  	BLOCKS=`du -bks ${HDDDIR} | cut -f 1`
> -	SIZE=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
> +	BLOCKS=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
> +
> +	# Ensure total sectors is an integral number of sectors per
> +	# track or mcopy will complain. Sectors are 512 bytes, and we
> +	# generate images with 32 sectors per track. This calculation is
> +	# done in blocks, thus the mod by 16 instead of 32.
> +	BLOCKS=$(expr $BLOCKS + $(expr 16 - $(expr $BLOCKS % 16)))
> +
> +	mkdosfs -n ${BOOTDD_VOLUME_ID} -S 512 -C ${HDDIMG} $BLOCKS 
> +	mcopy -i ${HDDIMG} -s ${HDDDIR}/* ::/
>  
> -	mkdosfs -n ${BOOTDD_VOLUME_ID} -d ${HDDDIR} -C ${HDDIMG} $SIZE 
>  	syslinux ${HDDIMG}
>  	chmod 644 ${HDDIMG}
>  
>  	ROOTFSBLOCKS=`du -Lbks ${ROOTFS} | cut -f 1`
> -	TOTALSIZE=`expr $SIZE + $ROOTFSBLOCKS`
> -	END1=`expr $SIZE \* 1024`
> +	TOTALSIZE=`expr $BLOCKS + $ROOTFSBLOCKS`
> +	END1=`expr $BLOCKS \* 1024`
>  	END2=`expr $END1 + 512`
>  	END3=`expr \( $ROOTFSBLOCKS \* 1024 \) + $END1`
>  

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

[Build App RFC 4/5] vmdk: Update for direstdisk

[Saul Wold]

Signed-off-by: Saul Wold <sgw@linux.intel.com>
---
 meta/classes/image-vmdk.bbclass |   15 ++++++++++++---
 1 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/meta/classes/image-vmdk.bbclass b/meta/classes/image-vmdk.bbclass
index 9d596cd..736d1d7 100644
--- a/meta/classes/image-vmdk.bbclass
+++ b/meta/classes/image-vmdk.bbclass
@@ -1,15 +1,23 @@
 
 NOISO = "1"
+
+SYSLINUX_ROOT = "root=/dev/hda2 "
 SYSLINUX_PROMPT = "0"
 SYSLINUX_TIMEOUT = "1"
 SYSLINUX_LABELS = "boot"
+LABELS_append = " ${SYSLINUX_LABELS} "
+
+# need to define the dependency and the ROOTFS for directdisk
+do_bootdirectdisk[depends] += "${IMAGE_BASENAME}:do_rootfs"
+ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_BASENAME}-${MACHINE}.ext3"
 
 # creating VMDK relies on having a live hddimg so ensure we
 # inherit it here.
-inherit image-live
+#inherit image-live
+inherit boot-directdisk
 
 create_vmdk_image () {
-	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
+	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
 	ln -s ${IMAGE_NAME}.vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.vmdk
 
 }
@@ -18,7 +26,8 @@ python do_vmdkimg() {
         bb.build.exec_func('create_vmdk_image', d)
 }
 
-addtask vmdkimg after do_bootimg before do_build
+#addtask vmdkimg after do_bootimg before do_build
+addtask vmdkimg after do_bootdirectdisk before do_build
 do_vmdkimg[nostamp] = "1"
 
 do_vmdkimg[depends] += "qemu-native:do_populate_sysroot" 
-- 
1.7.7.6

Re: [Build App RFC 4/5] vmdk: Update for direstdisk

[Darren Hart]

The conversion to directdisk is necessary due to the limitation in the
mkdosfs version we carry and larger filesystems. Please note in the
commit message so readers can know WHY the change was made.


On 03/25/2012 11:47 PM, Saul Wold wrote:
> Signed-off-by: Saul Wold <sgw@linux.intel.com>
> ---
>  meta/classes/image-vmdk.bbclass |   15 ++++++++++++---
>  1 files changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/image-vmdk.bbclass b/meta/classes/image-vmdk.bbclass
> index 9d596cd..736d1d7 100644
> --- a/meta/classes/image-vmdk.bbclass
> +++ b/meta/classes/image-vmdk.bbclass
> @@ -1,15 +1,23 @@
>  
>  NOISO = "1"
> +
> +SYSLINUX_ROOT = "root=/dev/hda2 "
>  SYSLINUX_PROMPT = "0"
>  SYSLINUX_TIMEOUT = "1"
>  SYSLINUX_LABELS = "boot"
> +LABELS_append = " ${SYSLINUX_LABELS} "
> +
> +# need to define the dependency and the ROOTFS for directdisk
> +do_bootdirectdisk[depends] += "${IMAGE_BASENAME}:do_rootfs"
> +ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_BASENAME}-${MACHINE}.ext3"
>  
>  # creating VMDK relies on having a live hddimg so ensure we
>  # inherit it here.
> -inherit image-live
> +#inherit image-live


A development artifact I presume? No need to leave the old image type
commented out. Should probably also remove the comment as with
directdisk that is no longer the case - right?


> +inherit boot-directdisk
>  
>  create_vmdk_image () {
> -	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
> +	qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk
>  	ln -s ${IMAGE_NAME}.vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.vmdk
>  
>  }
> @@ -18,7 +26,8 @@ python do_vmdkimg() {
>          bb.build.exec_func('create_vmdk_image', d)
>  }
>  
> -addtask vmdkimg after do_bootimg before do_build
> +#addtask vmdkimg after do_bootimg before do_build

Ditto.

> +addtask vmdkimg after do_bootdirectdisk before do_build
>  do_vmdkimg[nostamp] = "1"
>  
>  do_vmdkimg[depends] += "qemu-native:do_populate_sysroot" 

Thanks,

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

[Build App RFC 5/5] builder: Enable auto starting of Hob

[Saul Wold]

Signed-off-by: Saul Wold <sgw@linux.intel.com>
---
 meta/recipes-graphics/builder/builder_0.1.bb       |    2 +-
 .../builder/files/builder_hob_start.sh             |   10 +++++++---
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-graphics/builder/builder_0.1.bb b/meta/recipes-graphics/builder/builder_0.1.bb
index ceb9612..1324e5b 100644
--- a/meta/recipes-graphics/builder/builder_0.1.bb
+++ b/meta/recipes-graphics/builder/builder_0.1.bb
@@ -1,7 +1,7 @@
 SUMMARY = "New user to do specific job"
 DESCRIPTION = "This recipe create a new user named ${PN}, who is used for specific jobs like building. The task can be auto started via mini X"
 SECTION = "x11"
-PR = "r0"
+PR = "r1"
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://builder_hob_start.sh;endline=5;md5=84796c3c41785d86100fdabcbdade00e"
 
diff --git a/meta/recipes-graphics/builder/files/builder_hob_start.sh b/meta/recipes-graphics/builder/files/builder_hob_start.sh
index fdafc4e..f43e6c5 100644
--- a/meta/recipes-graphics/builder/files/builder_hob_start.sh
+++ b/meta/recipes-graphics/builder/files/builder_hob_start.sh
@@ -4,8 +4,12 @@
 #be put here
 
 # start hob here
-#cd /intel/poky/poky
-#. ./oe-init-build-env
-#../scripts/hob
+export PSEUDO_PREFIX=/usr
+export PSEUDO_LOCALSTATEDIR=/home/builder/pseudo
+export PSEUDO_LIBDIR=/usr/lib/pseudo/lib64
+
+cd /home/builder/poky
+. ./oe-init-build-env
+hob &
 
 matchbox-terminal&
-- 
1.7.7.6