Re: core-image-tiny-initramfs failing to parse because of WIC (WKS_FULL_PATH, )

[Martin Jansa <martin.jansa@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 11673 bytes --]

On Wed, Feb 15, 2017 at 10:58:49AM +0200, Ed Bartosh wrote:
> Hi Martin,
> 
> Sorry for the delay. I missedd your message somehow.
> 
> I can't reproduce this. Tried with latest poky and latest oe-core +
> bitbake. I built core-image-tiny-initramfs from scratch and bitbake -e
> busybox works for me just fine.
> 
> Can you provide more details about your setup, please?

Hi Ed,

I was also having troubles reproducing it in different setup as well, until I've
found out that my local.conf had this line:

IMAGE_FSTYPES_qemux86 = "vmdk"

which prevented ever using wic (USING_WIC variable was always empty).

Now with this line removed I can share the complete exception reproduced
with just oe-core layer included:

$ bitbake -e core-image-tiny-initramfs | tee env.core-image-tiny-initramfs5
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 38869, PID: 21506
Parsing recipes...ERROR: ExpansionError during parsing /OE/build/oe-core/openembedded-core/meta/recipes-core/images/core-image-tiny-initramfs.bb
Traceback (most recent call last):
  File "/OE/build/oe-core/bitbake/lib/bb/data_smart.py", line 412, in DataSmart.expandWithRefs(s='\tout="${IMGDEPLOYDIR}/${IMAGE_NAME}"\n\twks="${WKS_FULL_PATH}"\n\tif [ -z "$wks" ]; then\n\t\tbbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."\n\tfi\n\n\tBUILDDIR="${TOPDIR}" wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$out/" ${WIC_CREATE_EXTRA_ARGS}\n\tmv "$out/$(basename "${wks%.wks}")"*.direct "$out${IMAGE_NAME_SUFFIX}.wic"\n\trm -rf "$out/"\n', varname='IMAGE_CMD'):
                 try:
    >                s = __expand_var_regexp__.sub(varparse.var_sub, s)
                     try:
  File "/OE/build/oe-core/bitbake/lib/bb/data_smart.py", line 111, in VariableParse.var_sub(match=<_sre.SRE_Match object; span=(43, 59), match='${WKS_FULL_PATH}'>):
                 else:
    >                var = self.d.getVarFlag(key, "_content")
                 self.references.add(key)
  File "/OE/build/oe-core/bitbake/lib/bb/data_smart.py", line 773, in DataSmart.getVarFlag(var='WKS_FULL_PATH', flag='_content', expand=True, noweakdefault=False, parsing=False):
                     cachename = var + "[" + flag + "]"
    >            value = self.expand(value, cachename)

  File "/OE/build/oe-core/bitbake/lib/bb/data_smart.py", line 436, in DataSmart.expand(s="${@wks_search('${WKS_FILES}'.split(), '${WKS_SEARCH_PATH}') or ''}", varname='WKS_FULL_PATH'):
         def expand(self, s, varname = None):
    >        return self.expandWithRefs(s, varname).value

  File "/OE/build/oe-core/bitbake/lib/bb/data_smart.py", line 426, in DataSmart.expandWithRefs(s="${@wks_search('directdisk.wks core-image-tiny-initramfs.wks'.split(), '/OE/build/oe-core/openembedded-core/meta/recipes-core/images:${@':'.join('%s/wic' % p for p in '/OE/build/oe-core:${TMPDIR}:/OE/build/oe-core/openembedded-core/meta'.split(':'))}:${@':'.join('%s/scripts/lib/wic/canned-wks' % l for l in '/OE/build/oe-core:${TMPDIR}:/OE/build/oe-core/openembedded-core/meta:/OE/build/oe-core/openembedded-core'.split(':'))}') or ''}", varname='WKS_FULL_PATH'):
                 except Exception as exc:
    >                raise ExpansionError(varname, s, exc) from exc

bb.data_smart.ExpansionError: Failure expanding variable WKS_FULL_PATH, expression was ${@wks_search('directdisk.wks core-image-tiny-initramfs.wks'.split(), '/OE/build/oe-core/openembedded-core/meta/recipes-core/images:${@':'.join('%s/wic' % p for p in '/OE/build/oe-core:${TMPDIR}:/OE/build/oe-core/openembedded-core/meta'.split(':'))}:${@':'.join('%s/scripts/lib/wic/canned-wks' % l for l in '/OE/build/oe-core:${TMPDIR}:/OE/build/oe-core/openembedded-core/meta:/OE/build/oe-core/openembedded-core'.split(':'))}') or ''} which triggered exception SyntaxError: invalid syntax (WKS_FULL_PATH, line 1)


Summary: There was 1 ERROR message shown, returning a non-zero exit code.

The strange part of this exception is the unexpanded ${TMPDIR} in the WKS_SEARCH_PATH value,
that lead me to following 2 lines in conf/bblayers.conf:

TMPDIR := "/OE/build/oe-core"
BBPATH = "${TOPDIR}:${TMPDIR}"

I'm not sure why it appears unexpanded in WKS_SEARCH_PATH, I see you do some magick with all these
WKS_* variables, especially in that anonymous python, but the exception is actually from
expanding IMAGE_CMD_wic () function and it goes away if I remove following line from it:

        wks="${WKS_FULL_PATH}"

And it also goes away if I use immediate expansion when setting that BBPATH in bblayers.conf.

Interesting variables from bitbake -e with immediate expansion:
# $BBPATH [4 operations]
#   immediate /OE/build/oe-core/conf/bblayers.conf:6
#     "${TOPDIR}:${TMPDIR}"
#   postdot /OE/build/oe-core/openembedded-core/meta/conf/layer.conf:2
#     ":${LAYERDIR}"
#   set data_smart.py:911 [expandVarref]
#     "/OE/build/oe-core:/OE/build/oe-core:/OE/build/oe-core/openembedded-core/meta"
#   set /OE/build/oe-core/openembedded-core/meta/conf/documentation.conf:94
#     [doc] "Used by BitBake to locate .bbclass and configuration files. This variable is analogous to the PATH variable."
# pre-expansion value:
#   "/OE/build/oe-core:/OE/build/oe-core:/OE/build/oe-core/openembedded-core/meta"
BBPATH="/OE/build/oe-core:/OE/build/oe-core:/OE/build/oe-core/openembedded-core/meta"

# $WKS_SEARCH_PATH
#   set? /OE/build/oe-core/openembedded-core/meta/classes/image_types_wic.bbclass:10
#     "${THISDIR}:${@':'.join('%s/wic' % p for p in '${BBPATH}'.split(':'))}:${@':'.join('%s/scripts/lib/wic/canned-wks' % l for l in '${BBPATH}:${COREBASE}'.split(':'))}"
WKS_SEARCH_PATH="/OE/build/oe-core/openembedded-core/meta/recipes-core/images:/OE/build/oe-core/wic:/OE/build/oe-core/wic:/OE/build/oe-core/openembedded-core/meta/wic:/OE/build/oe-core/scripts/lib/wic/canned-wks:/OE/build/oe-core/scripts/lib/wic/canned-wks:/OE/build/oe-core/openembedded-core/meta/scripts/lib/wic/canned-wks:/OE/build/oe-core/openembedded-core/scripts/lib/wic/canned-wks"

# $WKS_FULL_PATH
#   set /OE/build/oe-core/openembedded-core/meta/classes/image_types_wic.bbclass:11
#     "${@wks_search('${WKS_FILES}'.split(), '${WKS_SEARCH_PATH}') or ''}"
WKS_FULL_PATH="/OE/build/oe-core/openembedded-core/scripts/lib/wic/canned-wks/directdisk.wks"

And the same without immediate expansion, but with 'wks="${WKS_FULL_PATH}"' line removed so that it doesn't fail to parse:
# $BBPATH [4 operations]
#   set /OE/build/oe-core/conf/bblayers.conf:6
#     "${TOPDIR}:${TMPDIR}"
#   postdot /OE/build/oe-core/openembedded-core/meta/conf/layer.conf:2
#     ":${LAYERDIR}"
#   set data_smart.py:911 [expandVarref]
#     "${TOPDIR}:${TMPDIR}:/OE/build/oe-core/openembedded-core/meta"
#   set /OE/build/oe-core/openembedded-core/meta/conf/documentation.conf:94
#     [doc] "Used by BitBake to locate .bbclass and configuration files. This variable is analogous to the PATH variable."
# pre-expansion value:
#   "${TOPDIR}:${TMPDIR}:/OE/build/oe-core/openembedded-core/meta"
BBPATH="/OE/build/oe-core:/OE/build/oe-core-glibc:/OE/build/oe-core/openembedded-core/meta"

# $WKS_SEARCH_PATH
#   set? /OE/build/oe-core/openembedded-core/meta/classes/image_types_wic.bbclass:10
#     "${THISDIR}:${@':'.join('%s/wic' % p for p in '${BBPATH}'.split(':'))}:${@':'.join('%s/scripts/lib/wic/canned-wks' % l for l in '${BBPATH}:${COREBASE}'.split(':'))}"
WKS_SEARCH_PATH="/OE/build/oe-core/openembedded-core/meta/recipes-core/images:/OE/build/oe-core/wic:/OE/build/oe-core-glibc/wic:/OE/build/oe-core/openembedded-core/meta/wic:/OE/build/oe-core/scripts/lib/wic/canned-wks:/OE/build/oe-core-glibc/scripts/lib/wic/canned-wks:/OE/build/oe-core/openembedded-core/meta/scripts/lib/wic/canned-wks:/OE/build/oe-core/openembedded-core/scripts/lib/wic/canned-wks"

# $WKS_FULL_PATH
#   set /OE/build/oe-core/openembedded-core/meta/classes/image_types_wic.bbclass:11
#     "${@wks_search('${WKS_FILES}'.split(), '${WKS_SEARCH_PATH}') or ''}"
WKS_FULL_PATH="/OE/build/oe-core/openembedded-core/scripts/lib/wic/canned-wks/directdisk.wks"

Let me know if sharing "bitbake -e core-image-tiny-initramfs" from these 2 tests would be useful for you.

Regards,
> 
> On Thu, Feb 09, 2017 at 09:39:30AM +0100, Martin Jansa wrote:
> > Running e.g. bitbake -e busybox with latest oe-core triggers this badly
> > looking exception:
> > 
> > Can we add some weak assignment for people who don't care about wic
> > or core-image-tiny-initramfs to continue using oe-core?
> > 
> > ERROR: ExpansionError during parsing /OE/oe-core/meta/recipes-core/images/
> > core-image-tiny-initramfs.bb
> > Traceback (most recent call last):
> >   File "/OE/bitbake/lib/bb/data_smart.py", line 412, in
> > DataSmart.expandWithRefs(s='\tout="${IMGDEPLOYDIR}/${IMAGE_NAME}"\n\twks="${WKS_FULL_PATH}"\n\tif
> > [ -z "$wks" ]; then\n\t\tbbfatal "No kickstart files from WKS_FILES were
> > found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES
> > appropriately."\n\tfi\n\n\tBUILDDIR="${TOPDIR}" wic create "$wks" --vars
> > "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$out/"
> > ${WIC_CREATE_EXTRA_ARGS}\n\tmv "$out/$(basename "${wks%.wks}")"*.direct
> > "$out${IMAGE_NAME_SUFFIX}.wic"\n\trm -rf "$out/"\n', varname='IMAGE_CMD'):
> >                  try:
> >     >                s = __expand_var_regexp__.sub(varparse.var_sub, s)
> >                      try:
> >   File "/OE/bitbake/lib/bb/data_smart.py", line 111, in
> > VariableParse.var_sub(match=<_sre.SRE_Match object; span=(43, 59),
> > match='${WKS_FULL_PATH}'>):
> >                  else:
> >     >                var = self.d.getVarFlag(key, "_content")
> >                  self.references.add(key)
> >   File "/OE/bitbake/lib/bb/data_smart.py", line 773, in
> > DataSmart.getVarFlag(var='WKS_FULL_PATH', flag='_content', expand=True,
> > noweakdefault=False, parsing=False):
> >                      cachename = var + "[" + flag + "]"
> >     >            value = self.expand(value, cachename)
> > 
> >   File "/OE/bitbake/lib/bb/data_smart.py", line 436, in
> > DataSmart.expand(s="${@wks_search('${WKS_FILES}'.split(),
> > '${WKS_SEARCH_PATH}') or ''}", varname='WKS_FULL_PATH'):
> >          def expand(self, s, varname = None):
> >     >        return self.expandWithRefs(s, varname).value
> > 
> >   File "/OE/bitbake/lib/bb/data_smart.py", line 426, in
> > DataSmart.expandWithRefs(s="${@wks_search('directdisk.wks
> > core-image-tiny-initramfs.wks'.split(),
> > '/OE/oe-core/meta/recipes-core/images:${@':'.join('%s/wic' % p for p in
> > '/OE:${TMPDIR}:/OE/oe-core/meta'.split(':'))}:${@':'.join('%s/scripts/lib/wic/canned-wks'
> > % l for l in '/OE:${TMPDIR}:/OE/oe-core/meta:/OE/oe-core'.split(':'))}') or
> > ''}", varname='WKS_FULL_PATH'):
> >                  except Exception as exc:
> >     >                raise ExpansionError(varname, s, exc) from exc
> > 
> > bb.data_smart.ExpansionError: Failure expanding variable WKS_FULL_PATH,
> > expression was ${@wks_search('directdisk.wks
> > core-image-tiny-initramfs.wks'.split(),
> > '/OE/oe-core/meta/recipes-core/images:${@':'.join('%s/wic' % p for p in
> > '/OE:${TMPDIR}:/OE/oe-core/meta'.split(':'))}:${@':'.join('%s/scripts/lib/wic/canned-wks'
> > % l for l in '/OE:${TMPDIR}:/OE/oe-core/meta:/OE/oe-core'.split(':'))}') or
> > ''} which triggered exception SyntaxError: invalid syntax (WKS_FULL_PATH,
> > line 1)
> 
> --
> Regards,
> Ed

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]