[PATCH v2] gomod: extract license files for omitted modules

[PATCH v2] gomod: extract license files for omitted modules

[jeroen]

From: Jeroen Hofstee <jhofstee@victronenergy.com>

If a gomod is omitted with a PACKAGECONFIG option its license file
doesn't get extracted to the gomod cache dir and hence do_populate_lic
will complain that the license file isn't found. This adds a task
do_extract_lic after do_compile and before do_populate_lic to make
sure the license files are extracted in such a case.

Signed-off-by: Jeroen Hofstee <jhofstee@victronenergy.com>

---
v2:

- drop bb.utils.mkdirhier(os.path.dirname(cachefile)), extract
  will create it.
- fix typo in the commit msg, it is do_populate_lic
---
 meta/classes-recipe/go-mod.bbclass | 33 ++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/meta/classes-recipe/go-mod.bbclass b/meta/classes-recipe/go-mod.bbclass
index a15dda8f0e..4c0496e1e7 100644
--- a/meta/classes-recipe/go-mod.bbclass
+++ b/meta/classes-recipe/go-mod.bbclass
@@ -29,6 +29,39 @@ do_unpack[cleandirs] += "${GOMODCACHE}"
 GO_WORKDIR ?= "${GO_IMPORT}"
 do_compile[dirs] += "${B}/src/${GO_WORKDIR}"
 
+python do_extract_lic() {
+    import zipfile
+
+    lics = d.getVar("LIC_FILES_CHKSUM")
+    cache = d.getVar("GOMODCACHE")
+    dldir = os.path.join(cache, "cache", "download")
+    prefix = "file://pkg/mod/"
+
+    for lic in lics.split():
+        if not lic.startswith(prefix):
+            continue
+
+        try:
+            src = lic[len(prefix):].split(";")[0]
+            url, suffix = src.split("@v")
+            version, _, file = suffix.partition(os.path.sep)
+        except:
+            continue
+
+        cachefile = os.path.join(cache, src)
+        zip = os.path.join(dldir, url, "@v", "v" + version) + ".zip"
+        if os.path.exists(cachefile) or not os.path.exists(zip):
+            continue
+
+        try:
+            bb.note(f"extract {src} from {zip}")
+            zipfile.ZipFile(zip).extract(src, cache)
+        except:
+            bb.warn(f"could not extract {src} from {zip}")
+}
+
 # Make go install unpack the module zip files in the module cache directory
 # before the license directory is polulated with license files.
+# Do make sure licenses get extracted for omitted modules.
+addtask do_extract_lic after do_compile before do_populate_lic
 addtask do_compile before do_populate_lic
-- 
2.43.0

Re: [OE-core] [PATCH v2] gomod: extract license files for omitted modules

[Jeroen Hofstee]

Hi Christian, Ross or some else to whom it may concern ,

Can you also have a look at:
https://patchwork.yoctoproject.org/project/oe-core/patch/20250719150105.2697443-1-jeroen@myspectrum.nl/

It simply makes sure LICENSE files are always present.

Thanks,
Jeroen

On 7/19/25 17:01, Jeroen Hofstee wrote:
> From: Jeroen Hofstee <jhofstee@victronenergy.com>
>
> If a gomod is omitted with a PACKAGECONFIG option its license file
> doesn't get extracted to the gomod cache dir and hence do_populate_lic
> will complain that the license file isn't found. This adds a task
> do_extract_lic after do_compile and before do_populate_lic to make
> sure the license files are extracted in such a case.
>
> Signed-off-by: Jeroen Hofstee <jhofstee@victronenergy.com>
>
> ---
> v2:
>
> - drop bb.utils.mkdirhier(os.path.dirname(cachefile)), extract
>    will create it.
> - fix typo in the commit msg, it is do_populate_lic
> ---
>   meta/classes-recipe/go-mod.bbclass | 33 ++++++++++++++++++++++++++++++
>   1 file changed, 33 insertions(+)
>
> diff --git a/meta/classes-recipe/go-mod.bbclass b/meta/classes-recipe/go-mod.bbclass
> index a15dda8f0e..4c0496e1e7 100644
> --- a/meta/classes-recipe/go-mod.bbclass
> +++ b/meta/classes-recipe/go-mod.bbclass
> @@ -29,6 +29,39 @@ do_unpack[cleandirs] += "${GOMODCACHE}"
>   GO_WORKDIR ?= "${GO_IMPORT}"
>   do_compile[dirs] += "${B}/src/${GO_WORKDIR}"
>   
> +python do_extract_lic() {
> +    import zipfile
> +
> +    lics = d.getVar("LIC_FILES_CHKSUM")
> +    cache = d.getVar("GOMODCACHE")
> +    dldir = os.path.join(cache, "cache", "download")
> +    prefix = "file://pkg/mod/"
> +
> +    for lic in lics.split():
> +        if not lic.startswith(prefix):
> +            continue
> +
> +        try:
> +            src = lic[len(prefix):].split(";")[0]
> +            url, suffix = src.split("@v")
> +            version, _, file = suffix.partition(os.path.sep)
> +        except:
> +            continue
> +
> +        cachefile = os.path.join(cache, src)
> +        zip = os.path.join(dldir, url, "@v", "v" + version) + ".zip"
> +        if os.path.exists(cachefile) or not os.path.exists(zip):
> +            continue
> +
> +        try:
> +            bb.note(f"extract {src} from {zip}")
> +            zipfile.ZipFile(zip).extract(src, cache)
> +        except:
> +            bb.warn(f"could not extract {src} from {zip}")
> +}
> +
>   # Make go install unpack the module zip files in the module cache directory
>   # before the license directory is polulated with license files.
> +# Do make sure licenses get extracted for omitted modules.
> +addtask do_extract_lic after do_compile before do_populate_lic
>   addtask do_compile before do_populate_lic
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#220613): https://lists.openembedded.org/g/openembedded-core/message/220613
> Mute This Topic: https://lists.openembedded.org/mt/114238035/9720520
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [oe@myspectrum.nl]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2] gomod: extract license files for omitted modules

[Christian Lindeberg]

Hi,

An alternative more in line with conditional dependencies to C 
libraries, where
the license is pulled in if the dependency is added, would be to 
conditionally
add the licenses.

But in case this is not feasible I guess there is a need to unpack the extra
licenses as a workaround.

Some comments below.

Thanks,
Christian


On 02/09/2025 17:04, Jeroen Hofstee wrote:
> [You don't often get email from oe@myspectrum.nl. Learn why this is 
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Hi Christian, Ross or some else to whom it may concern ,
>
> Can you also have a look at:
> https://patchwork.yoctoproject.org/project/oe-core/patch/20250719150105.2697443-1-jeroen@myspectrum.nl/ 
>
>
> It simply makes sure LICENSE files are always present.
>
> Thanks,
> Jeroen
>
> On 7/19/25 17:01, Jeroen Hofstee wrote:
>> From: Jeroen Hofstee <jhofstee@victronenergy.com>
>>
>> If a gomod is omitted with a PACKAGECONFIG option its license file
>> doesn't get extracted to the gomod cache dir and hence do_populate_lic
>> will complain that the license file isn't found. This adds a task
>> do_extract_lic after do_compile and before do_populate_lic to make
>> sure the license files are extracted in such a case.
>>
>> Signed-off-by: Jeroen Hofstee <jhofstee@victronenergy.com>
>>
>> ---
>> v2:
>>
>> - drop bb.utils.mkdirhier(os.path.dirname(cachefile)), extract
>>    will create it.
>> - fix typo in the commit msg, it is do_populate_lic
>> ---
>>   meta/classes-recipe/go-mod.bbclass | 33 ++++++++++++++++++++++++++++++
>>   1 file changed, 33 insertions(+)
>>
>> diff --git a/meta/classes-recipe/go-mod.bbclass 
>> b/meta/classes-recipe/go-mod.bbclass
>> index a15dda8f0e..4c0496e1e7 100644
>> --- a/meta/classes-recipe/go-mod.bbclass
>> +++ b/meta/classes-recipe/go-mod.bbclass
>> @@ -29,6 +29,39 @@ do_unpack[cleandirs] += "${GOMODCACHE}"
>>   GO_WORKDIR ?= "${GO_IMPORT}"
>>   do_compile[dirs] += "${B}/src/${GO_WORKDIR}"
>>
>> +python do_extract_lic() {
>> +    import zipfile
>> +
>> +    lics = d.getVar("LIC_FILES_CHKSUM")
>> +    cache = d.getVar("GOMODCACHE")
>> +    dldir = os.path.join(cache, "cache", "download")
>> +    prefix = "file://pkg/mod/"
>> +
>> +    for lic in lics.split():
>> +        if not lic.startswith(prefix):
>> +            continue
>> +
>> +        try:
>> +            src = lic[len(prefix):].split(";")[0]
>> +            url, suffix = src.split("@v")
>> +            version, _, file = suffix.partition(os.path.sep)
The path separator in the URIs will not change with the OS but I guess 
this may
not be completely consistent across the code base.
>> +        except:
>> +            continue
>> +
>> +        cachefile = os.path.join(cache, src)
>> +        zip = os.path.join(dldir, url, "@v", "v" + version) + ".zip"
>> +        if os.path.exists(cachefile) or not os.path.exists(zip):
A non-existing zip file is unexpected and I think a warning is appropriate.
>> +            continue
>> +
>> +        try:
>> +            bb.note(f"extract {src} from {zip}")
>> +            zipfile.ZipFile(zip).extract(src, cache)
>> +        except:
>> +            bb.warn(f"could not extract {src} from {zip}")
>> +}
>> +
>>   # Make go install unpack the module zip files in the module cache 
>> directory
>>   # before the license directory is polulated with license files.
>> +# Do make sure licenses get extracted for omitted modules.
>> +addtask do_extract_lic after do_compile before do_populate_lic
>>   addtask do_compile before do_populate_lic
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#220613): 
>> https://lists.openembedded.org/g/openembedded-core/message/220613
>> Mute This Topic: https://lists.openembedded.org/mt/114238035/9720520
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
>> [oe@myspectrum.nl]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>