* [OE-core][PATCH] wic/engine: fix copying directories into wic image with ext* partition
@ 2025-10-03 20:31 Dragomir, Daniel
2025-10-06 20:08 ` Ross Burton
0 siblings, 1 reply; 5+ messages in thread
From: Dragomir, Daniel @ 2025-10-03 20:31 UTC (permalink / raw)
To: openembedded-core
wic uses debugfs to write on ext* partitions, but debugfs can only
write to the current working directory and it cannot copy complete
directory trees. Running 'wic ls' on a copied directory show this:
-l: Ext2 inode is not a directory
Fix this by creating a command list for debugfs (-f parameter) when
recursive parsing the host directory in order to create a similar
directory structure (mkdir) and copy files (write) on each level
into the destination directory from the wic's ext* partition.
Signed-off-by: Daniel Dragomir <daniel.dragomir@windriver.com>
---
scripts/lib/wic/engine.py | 63 ++++++++++++++++++++++++++++++---------
1 file changed, 49 insertions(+), 14 deletions(-)
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index b9e60cbe4e..9d596be3a7 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -345,29 +345,64 @@ class Disk:
path))
def copy(self, src, dest):
- """Copy partition image into wic image."""
- pnum = dest.part if isinstance(src, str) else src.part
+ """Copy files or directories to/from the vfat or ext* partition."""
+ pnum = dest.part if isinstance(src, str) else src.part
+ partimg = self._get_part_image(pnum)
if self.partitions[pnum].fstype.startswith('ext'):
- if isinstance(src, str):
- cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\
- format(os.path.dirname(dest.path), src, os.path.basename(src),
- self.debugfs, self._get_part_image(pnum))
- else: # copy from wic
- # run both dump and rdump to support both files and directory
+ if isinstance(src, str): # host to image case
+ if os.path.isdir(src):
+ base = os.path.abspath(src)
+ base_parent = os.path.dirname(base)
+ cmds = []
+ made = set()
+
+ for root, dirs, files in os.walk(base):
+ for fname in files:
+ host_file = os.path.join(root, fname)
+ rel = os.path.relpath(host_file, base_parent)
+ dest_file = os.path.join(dest.path, rel)
+ dest_dir = os.path.dirname(dest_file)
+
+ # create dir structure (mkdir -p)
+ parts = dest_dir.strip('/').split('/')
+ cur = ''
+ for p in parts:
+ cur = cur + '/' + p
+ if cur not in made:
+ cmds.append(f'mkdir "{cur}"')
+ made.add(cur)
+
+ cmds.append(f'write "{host_file}" "{dest_file}"')
+
+ # write script to a temp file
+ with tempfile.NamedTemporaryFile(mode='w', delete=False,
+ prefix='wic-debugfs-') as tf:
+ for line in cmds:
+ tf.write(line + '\n')
+ scriptname = tf.name
+
+ cmd = f"{self.debugfs} -w -f {scriptname} {partimg}"
+
+ else: # single file
+ cmd = "printf 'cd {}\nwrite {} {}\n' | {} -w {}".\
+ format(os.path.dirname(dest.path), src,
+ os.path.basename(src), self.debugfs, partimg)
+
+ else: # image to host case
cmd = "printf 'cd {}\ndump /{} {}\nrdump /{} {}\n' | {} {}".\
format(os.path.dirname(src.path), src.path,
- dest, src.path, dest, self.debugfs,
- self._get_part_image(pnum))
+ dest, src.path, dest, self.debugfs, partimg)
+
else: # fat
if isinstance(src, str):
cmd = "{} -i {} -snop {} ::{}".format(self.mcopy,
- self._get_part_image(pnum),
- src, dest.path)
+ partimg,
+ src, dest.path)
else:
cmd = "{} -i {} -snop ::{} {}".format(self.mcopy,
- self._get_part_image(pnum),
- src.path, dest)
+ partimg,
+ src.path, dest)
exec_cmd(cmd, as_shell=True)
self._put_part_image(pnum)
--
2.25.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [OE-core][PATCH] wic/engine: fix copying directories into wic image with ext* partition
2025-10-03 20:31 [OE-core][PATCH] wic/engine: fix copying directories into wic image with ext* partition Dragomir, Daniel
@ 2025-10-06 20:08 ` Ross Burton
2025-12-11 10:07 ` Richard Purdie
2025-12-17 13:15 ` Dragomir, Daniel
0 siblings, 2 replies; 5+ messages in thread
From: Ross Burton @ 2025-10-06 20:08 UTC (permalink / raw)
To: daniel.dragomir@windriver.com; +Cc: openembedded-core@lists.openembedded.org
On 3 Oct 2025, at 21:31, Dragomir, Daniel via lists.openembedded.org <daniel.dragomir=windriver.com@lists.openembedded.org> wrote:
>
> wic uses debugfs to write on ext* partitions, but debugfs can only
> write to the current working directory and it cannot copy complete
> directory trees. Running 'wic ls' on a copied directory show this:
> -l: Ext2 inode is not a directory
>
> Fix this by creating a command list for debugfs (-f parameter) when
> recursive parsing the host directory in order to create a similar
> directory structure (mkdir) and copy files (write) on each level
> into the destination directory from the wic's ext* partition.
This is great, but could you please add a test case to the wic selftests to exercise this and ensure it doesn’t break in the future?
Thanks,
Ross
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core][PATCH] wic/engine: fix copying directories into wic image with ext* partition
2025-10-06 20:08 ` Ross Burton
@ 2025-12-11 10:07 ` Richard Purdie
2025-12-17 13:15 ` Dragomir, Daniel
1 sibling, 0 replies; 5+ messages in thread
From: Richard Purdie @ 2025-12-11 10:07 UTC (permalink / raw)
To: ross.burton, daniel.dragomir@windriver.com
Cc: openembedded-core@lists.openembedded.org
On Mon, 2025-10-06 at 20:08 +0000, Ross Burton via
lists.openembedded.org wrote:
> On 3 Oct 2025, at 21:31, Dragomir, Daniel via lists.openembedded.org
> <daniel.dragomir=windriver.com@lists.openembedded.org> wrote:
> >
> > wic uses debugfs to write on ext* partitions, but debugfs can only
> > write to the current working directory and it cannot copy complete
> > directory trees. Running 'wic ls' on a copied directory show this:
> > -l: Ext2 inode is not a directory
> >
> > Fix this by creating a command list for debugfs (-f parameter) when
> > recursive parsing the host directory in order to create a similar
> > directory structure (mkdir) and copy files (write) on each level
> > into the destination directory from the wic's ext* partition.
>
> This is great, but could you please add a test case to the wic
> selftests to exercise this and ensure it doesn’t break in the future?
This has been waiting on the tests for a while in master-next. Is that
something you are able to do?
Cheers,
Richard
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core][PATCH] wic/engine: fix copying directories into wic image with ext* partition
2025-10-06 20:08 ` Ross Burton
2025-12-11 10:07 ` Richard Purdie
@ 2025-12-17 13:15 ` Dragomir, Daniel
2025-12-17 14:48 ` Richard Purdie
1 sibling, 1 reply; 5+ messages in thread
From: Dragomir, Daniel @ 2025-12-17 13:15 UTC (permalink / raw)
To: Ross Burton
Cc: openembedded-core@lists.openembedded.org, Kexin.Hao,
mathieu.dubois-briand
[-- Attachment #1: Type: text/plain, Size: 2715 bytes --]
Sorry for the late response... I’m working this week on a selftest for
the case fixed by this patch.
Meanwhile, I was focused on another aspect related to this patch. Our
client reported different behavior when copying complex directory
structures onto a WIC image using different servers.
It appears that the issue is related to the debugfs version
(e2fsprogs package).
On Ubuntu 24.04 with e2fsprogs 4.17, all files are copied correctly,
while on Ubuntu 20.04 with e2fsprogs 1.45, the copy operation fails to
copy all files and does not create all directories. This failure is
silent, with no error reported by debugfs.
I tried several approaches to solve this (splitting the command list in
max 100 instances and calling debugfs 1.45 multiple times), but without
success.
For our client, we decided to add a debugfs version check in wic code
and to add a warning if the version is < 1.47.
On those specific servers, they will use wic with the
-n/--native-sysroot option, pointing to a scarthgap SDK that contains
1.47 version of the e2fsprogs package.
How should we handle this exception? This was fixed in newer debugfs
version, but should we add a debugfs version warning when broken version
of debugfs is used on host?
Regards,
Daniel
------------------------------------------------------------------------
*From:* Ross Burton <Ross.Burton@arm.com>
*Sent:* Monday, October 6, 2025 11:08 PM
*To:* Dragomir, Daniel <Daniel.Dragomir@windriver.com>
*Cc:* openembedded-core@lists.openembedded.org
<openembedded-core@lists.openembedded.org>
*Subject:* Re: [OE-core][PATCH] wic/engine: fix copying directories into
wic image with ext* partition
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender
and know the content is safe.
On 3 Oct 2025, at 21:31, Dragomir, Daniel via lists.openembedded.org
<daniel.dragomir=windriver.com@lists.openembedded.org> wrote:
>
> wic uses debugfs to write on ext* partitions, but debugfs can only
> write to the current working directory and it cannot copy complete
> directory trees. Running 'wic ls' on a copied directory show this:
> -l: Ext2 inode is not a directory
>
> Fix this by creating a command list for debugfs (-f parameter) when
> recursive parsing the host directory in order to create a similar
> directory structure (mkdir) and copy files (write) on each level
> into the destination directory from the wic's ext* partition.
This is great, but could you please add a test case to the wic selftests
to exercise this and ensure it doesn’t break in the future?
Thanks,
Ross
[-- Attachment #2: Type: text/html, Size: 7693 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core][PATCH] wic/engine: fix copying directories into wic image with ext* partition
2025-12-17 13:15 ` Dragomir, Daniel
@ 2025-12-17 14:48 ` Richard Purdie
0 siblings, 0 replies; 5+ messages in thread
From: Richard Purdie @ 2025-12-17 14:48 UTC (permalink / raw)
To: daniel.dragomir, Ross Burton
Cc: openembedded-core@lists.openembedded.org, Kexin.Hao,
mathieu.dubois-briand
On Wed, 2025-12-17 at 15:15 +0200, Dragomir, Daniel via lists.openembedded.org wrote:
> How should we handle this exception? This was fixed in newer debugfs version, but should we add a debugfs version warning when broken version of debugfs is used on host?
>
>
Thanks for the selftest, that helps a lot.
For the wic issue, I think a version check inside wic would be fine.
Please just make it clear in the commit message this is for standalone
use and we don't see this in OE since our version is new enough.
Cheers,
Richard
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2025-12-17 14:49 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-10-03 20:31 [OE-core][PATCH] wic/engine: fix copying directories into wic image with ext* partition Dragomir, Daniel
2025-10-06 20:08 ` Ross Burton
2025-12-11 10:07 ` Richard Purdie
2025-12-17 13:15 ` Dragomir, Daniel
2025-12-17 14:48 ` Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.