[PATCH v2] iotest: Unbreak 302 with python 3.13

[PATCH v2] iotest: Unbreak 302 with python 3.13

[Nir Soffer]

This test depends on TarFile.addfile() to add tar member header without
writing the member data, which we write ourself using qemu-nbd. Python
3.13 changed the function in a backward incompatible way[1] to require a
file object for tarinfo with non-zero size, breaking the test:

     -[{"name": "vm.ovf", "offset": 512, "size": 6}, {"name": "disk", "offset": 1536, "size": 393216}]
     +Traceback (most recent call last):
     +  File "/home/stefanha/qemu/tests/qemu-iotests/302", line 118, in <module>
     +    tar.addfile(disk)
     +    ~~~~~~~~~~~^^^^^^
     +  File "/usr/lib64/python3.13/tarfile.py", line 2262, in addfile
     +    raise ValueError("fileobj not provided for non zero-size regular file")
     +ValueError: fileobj not provided for non zero-size regular file

The new behavior makes sense for most users, but breaks our unusual
usage. Fix the test to add the member header directly using public but
undocumented attributes. This is more fragile but the test works again.

This also fixes a bug in the previous code - when calling addfile()
without a fileobject, tar.offset points to the start of the member data
instead of the end.

[1] https://github.com/python/cpython/pull/117988

Signed-off-by: Nir Soffer <nirsof@gmail.com>
---
 tests/qemu-iotests/302 | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/302 b/tests/qemu-iotests/302
index a6d79e727b..e980ec513f 100755
--- a/tests/qemu-iotests/302
+++ b/tests/qemu-iotests/302
@@ -115,13 +115,22 @@ with tarfile.open(tar_file, "w") as tar:
 
     disk = tarfile.TarInfo("disk")
     disk.size = actual_size
-    tar.addfile(disk)
 
-    # 6. Shrink the tar to the actual size, aligned to 512 bytes.
+    # Since python 3.13 we cannot use addfile() to create the member header.
+    # Add the tarinfo directly using public but undocumented attributes.
 
-    tar_size = offset + (disk.size + 511) & ~511
-    tar.fileobj.seek(tar_size)
-    tar.fileobj.truncate(tar_size)
+    buf = disk.tobuf(tar.format, tar.encoding, tar.errors)
+    tar.fileobj.write(buf)
+    tar.members.append(disk)
+
+    # Update the offset and position to the location of the next member.
+
+    tar.offset = offset + (disk.size + 511) & ~511
+    tar.fileobj.seek(tar.offset)
+
+    # 6. Shrink the tar to the actual size.
+
+    tar.fileobj.truncate(tar.offset)
 
 with tarfile.open(tar_file) as tar:
     members = [{"name": m.name, "size": m.size, "offset": m.offset_data}
-- 
2.39.5 (Apple Git-154)

Re: [PATCH v2] iotest: Unbreak 302 with python 3.13

[Eric Blake]

On Fri, Feb 28, 2025 at 09:57:08PM +0200, Nir Soffer wrote:
> This test depends on TarFile.addfile() to add tar member header without
> writing the member data, which we write ourself using qemu-nbd. Python
> 3.13 changed the function in a backward incompatible way[1] to require a
> file object for tarinfo with non-zero size, breaking the test:
> 
>      -[{"name": "vm.ovf", "offset": 512, "size": 6}, {"name": "disk", "offset": 1536, "size": 393216}]
>      +Traceback (most recent call last):
>      +  File "/home/stefanha/qemu/tests/qemu-iotests/302", line 118, in <module>
>      +    tar.addfile(disk)
>      +    ~~~~~~~~~~~^^^^^^
>      +  File "/usr/lib64/python3.13/tarfile.py", line 2262, in addfile
>      +    raise ValueError("fileobj not provided for non zero-size regular file")
>      +ValueError: fileobj not provided for non zero-size regular file
> 
> The new behavior makes sense for most users, but breaks our unusual
> usage. Fix the test to add the member header directly using public but
> undocumented attributes. This is more fragile but the test works again.
> 
> This also fixes a bug in the previous code - when calling addfile()
> without a fileobject, tar.offset points to the start of the member data
> instead of the end.
> 
> [1] https://github.com/python/cpython/pull/117988
> 
> Signed-off-by: Nir Soffer <nirsof@gmail.com>


Reviewed-by: Eric Blake <eblake@redhat.com>

> ---
>  tests/qemu-iotests/302 | 19 ++++++++++++++-----
>  1 file changed, 14 insertions(+), 5 deletions(-)
> 
> diff --git a/tests/qemu-iotests/302 b/tests/qemu-iotests/302
> index a6d79e727b..e980ec513f 100755
> --- a/tests/qemu-iotests/302
> +++ b/tests/qemu-iotests/302
> @@ -115,13 +115,22 @@ with tarfile.open(tar_file, "w") as tar:
>  
>      disk = tarfile.TarInfo("disk")
>      disk.size = actual_size
> -    tar.addfile(disk)
>  
> -    # 6. Shrink the tar to the actual size, aligned to 512 bytes.
> +    # Since python 3.13 we cannot use addfile() to create the member header.
> +    # Add the tarinfo directly using public but undocumented attributes.
>  
> -    tar_size = offset + (disk.size + 511) & ~511
> -    tar.fileobj.seek(tar_size)
> -    tar.fileobj.truncate(tar_size)
> +    buf = disk.tobuf(tar.format, tar.encoding, tar.errors)
> +    tar.fileobj.write(buf)
> +    tar.members.append(disk)
> +
> +    # Update the offset and position to the location of the next member.
> +
> +    tar.offset = offset + (disk.size + 511) & ~511
> +    tar.fileobj.seek(tar.offset)
> +
> +    # 6. Shrink the tar to the actual size.
> +
> +    tar.fileobj.truncate(tar.offset)
>  
>  with tarfile.open(tar_file) as tar:
>      members = [{"name": m.name, "size": m.size, "offset": m.offset_data}
> -- 
> 2.39.5 (Apple Git-154)
> 
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: [PATCH v2] iotest: Unbreak 302 with python 3.13

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

On Fri, Feb 28, 2025 at 09:57:08PM +0200, Nir Soffer wrote:
> This test depends on TarFile.addfile() to add tar member header without
> writing the member data, which we write ourself using qemu-nbd. Python
> 3.13 changed the function in a backward incompatible way[1] to require a
> file object for tarinfo with non-zero size, breaking the test:
> 
>      -[{"name": "vm.ovf", "offset": 512, "size": 6}, {"name": "disk", "offset": 1536, "size": 393216}]
>      +Traceback (most recent call last):
>      +  File "/home/stefanha/qemu/tests/qemu-iotests/302", line 118, in <module>
>      +    tar.addfile(disk)
>      +    ~~~~~~~~~~~^^^^^^
>      +  File "/usr/lib64/python3.13/tarfile.py", line 2262, in addfile
>      +    raise ValueError("fileobj not provided for non zero-size regular file")
>      +ValueError: fileobj not provided for non zero-size regular file
> 
> The new behavior makes sense for most users, but breaks our unusual
> usage. Fix the test to add the member header directly using public but
> undocumented attributes. This is more fragile but the test works again.
> 
> This also fixes a bug in the previous code - when calling addfile()
> without a fileobject, tar.offset points to the start of the member data
> instead of the end.
> 
> [1] https://github.com/python/cpython/pull/117988
> 
> Signed-off-by: Nir Soffer <nirsof@gmail.com>
> ---
>  tests/qemu-iotests/302 | 19 ++++++++++++++-----
>  1 file changed, 14 insertions(+), 5 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2] iotest: Unbreak 302 with python 3.13

[Eric Blake]

On Mon, Mar 03, 2025 at 04:41:56PM +0800, Stefan Hajnoczi wrote:
> On Fri, Feb 28, 2025 at 09:57:08PM +0200, Nir Soffer wrote:
> > This test depends on TarFile.addfile() to add tar member header without
> > writing the member data, which we write ourself using qemu-nbd. Python
> > 3.13 changed the function in a backward incompatible way[1] to require a
> > file object for tarinfo with non-zero size, breaking the test:
> > 
> >      -[{"name": "vm.ovf", "offset": 512, "size": 6}, {"name": "disk", "offset": 1536, "size": 393216}]
> >      +Traceback (most recent call last):
> >      +  File "/home/stefanha/qemu/tests/qemu-iotests/302", line 118, in <module>
> >      +    tar.addfile(disk)
> >      +    ~~~~~~~~~~~^^^^^^
> >      +  File "/usr/lib64/python3.13/tarfile.py", line 2262, in addfile
> >      +    raise ValueError("fileobj not provided for non zero-size regular file")
> >      +ValueError: fileobj not provided for non zero-size regular file
> > 
> > The new behavior makes sense for most users, but breaks our unusual
> > usage. Fix the test to add the member header directly using public but
> > undocumented attributes. This is more fragile but the test works again.
> > 
> > This also fixes a bug in the previous code - when calling addfile()
> > without a fileobject, tar.offset points to the start of the member data
> > instead of the end.
> > 
> > [1] https://github.com/python/cpython/pull/117988
> > 
> > Signed-off-by: Nir Soffer <nirsof@gmail.com>
> > ---
> >  tests/qemu-iotests/302 | 19 ++++++++++++++-----
> >  1 file changed, 14 insertions(+), 5 deletions(-)
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

Added to my NBD queue.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org