[PATCH] erofs-utils: tar: support archives without end-of-archive entry

[PATCH] erofs-utils: tar: support archives without end-of-archive entry

[Ivan Mikheykin]

Tar standard https://www.gnu.org/software/tar/manual/html_node/Standard.html
says that archive "terminated by an end-of-archive entry,
which consists of two 512 blocks of zero bytes".

Is also says:

"A reasonable system should write such end-of-file marker at the end
of an archive, but must not assume that such a block exists when
reading an archive. In particular, GNU tar does not treat missing
end-of-file marker as an error and silently ignores the fact."

It is rare for erofs to encounter such problem, as images are mostly
built with docker or buildah. But if you create image using tar library
in Golang directly uploading layers to registry, you'll get tar layers
without end-of-archive block. Running containers with such images will
trigger this error during extraction:

mkfs.erofs --tar=f --aufs --quiet -Enoinline_data test.erofs test-no-end.tar
<E> erofs: failed to read header block @ 42496
<E> erofs: 	Could not format the device : [Error 5] Input/output error

This patch fixes the problem by assuming that eof is equal to the end-of-archive.

Reproducible tar without end-of-archive (base64-encoded gzipped blob):
H4sICKVi2mgAA3Rlc3QtMTAtMi1ibG9ja3MudGFyAAtzDQr29PdjoCUwAAIzExMwbW5mCqYN
jQzANBgYGTEYmhqYmpqamRoaGTMYGBqaGJkyKBjQ1FVQUFpcklikoMCQkpmYll9ahFNdYkpu
Zh49HERfYKhnoWdowGVkYGSqa2Cua2jKNdAuGgX0BADwFwqsAAQAAA==

Signed-off-by: Ivan Mikheykin <ivan.mikheykin@flant.com>
---
 lib/tar.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/tar.c b/lib/tar.c
index 72c12ed..128f8b0 100644
--- a/lib/tar.c
+++ b/lib/tar.c
@@ -740,7 +740,7 @@ restart:
 	tar_offset = tar->offset;
 	ret = erofs_iostream_read(&tar->ios, (void **)&th, sizeof(*th));
 	if (ret != sizeof(*th)) {
-		if (tar->headeronly_mode || tar->ddtaridx_mode) {
+		if (tar->headeronly_mode || tar->ddtaridx_mode || tar->ios.feof) {
 			ret = 1;
 			goto out;
 		}
-- 
2.39.3 (Apple Git-146)

Re: [PATCH] erofs-utils: tar: support archives without end-of-archive entry

[Gao Xiang]

Hi Ivan,

On Mon, Sep 29, 2025 at 04:32:22PM +0300, Ivan Mikheykin wrote:
> Tar standard https://www.gnu.org/software/tar/manual/html_node/Standard.html
> says that archive "terminated by an end-of-archive entry,
> which consists of two 512 blocks of zero bytes".
> 
> Is also says:
> 
> "A reasonable system should write such end-of-file marker at the end
> of an archive, but must not assume that such a block exists when
> reading an archive. In particular, GNU tar does not treat missing
> end-of-file marker as an error and silently ignores the fact."
> 
> It is rare for erofs to encounter such problem, as images are mostly
> built with docker or buildah. But if you create image using tar library
> in Golang directly uploading layers to registry, you'll get tar layers
> without end-of-archive block. Running containers with such images will
> trigger this error during extraction:
> 
> mkfs.erofs --tar=f --aufs --quiet -Enoinline_data test.erofs test-no-end.tar
> <E> erofs: failed to read header block @ 42496
> <E> erofs: 	Could not format the device : [Error 5] Input/output error
> 
> This patch fixes the problem by assuming that eof is equal to the end-of-archive.
> 
> Reproducible tar without end-of-archive (base64-encoded gzipped blob):
> H4sICKVi2mgAA3Rlc3QtMTAtMi1ibG9ja3MudGFyAAtzDQr29PdjoCUwAAIzExMwbW5mCqYN
> jQzANBgYGTEYmhqYmpqamRoaGTMYGBqaGJkyKBjQ1FVQUFpcklikoMCQkpmYll9ahFNdYkpu
> Zh49HERfYKhnoWdowGVkYGSqa2Cua2jKNdAuGgX0BADwFwqsAAQAAA==

Thanks for the patch!

Could you confirm how docker/containerd or podman parses such image?

Because the POSIX standard says:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html
"At the end of the archive file there shall be two 512-byte blocks filled
with binary zeros, interpreted as an end-of-archive indicator."

So such tar layers will be non-standard, I wonder we need at least
a erofs_warn() message for such tars at least.

Thanks,
Gao Xiang

> 
> Signed-off-by: Ivan Mikheykin <ivan.mikheykin@flant.com>

Re: [PATCH] erofs-utils: tar: support archives without end-of-archive entry

[Ivan Mikheykin]

Hello!

On 9/29/25 5:50 PM, Gao Xiang wrote:

> Could you confirm how docker/containerd or podman parses such image?

I confirm that images with such layers work well at least in containerd 
with overlayfs. We encounter problems during experiments with containerd 
and erofs. Also, GNU tar and BSD tar are good without end-of-archive zeros.

> 
> Because the POSIX standard says:
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html
> "At the end of the archive file there shall be two 512-byte blocks filled
> with binary zeros, interpreted as an end-of-archive indicator."
> 
> So such tar layers will be non-standard, I wonder we need at least
> a erofs_warn() message for such tars at least.
> 

Good point. I think I can add erofs_warn() message to the patch.


P.S. I've noticed an interesting detail after submitting the patch. 
Produced erofs image reports an enormous file size after conversion error:

$ mkfs.erofs --tar=f --aufs --quiet -Enoinline_data test.erofs 
test-no-end.tar
-rw-r--r--    1 user  admin  2199023255552 Sep 29 13:43 test.erofs
$ du -sh test.erofs
4,0K    test.erofs

-- 

Re: [PATCH] erofs-utils: tar: support archives without end-of-archive entry

[Gao Xiang]

On 2025/9/30 15:42, Ivan Mikheykin wrote:
> Hello!
> 
> On 9/29/25 5:50 PM, Gao Xiang wrote:
> 
>> Could you confirm how docker/containerd or podman parses such image?
> 
> I confirm that images with such layers work well at least in containerd with overlayfs. We encounter problems during experiments with containerd and erofs. Also, GNU tar and BSD tar are good without end-of-archive zeros.
> 
>>
>> Because the POSIX standard says:
>> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html
>> "At the end of the archive file there shall be two 512-byte blocks filled
>> with binary zeros, interpreted as an end-of-archive indicator."
>>
>> So such tar layers will be non-standard, I wonder we need at least
>> a erofs_warn() message for such tars at least.
>>
> 
> Good point. I think I can add erofs_warn() message to the patch.

Yes, please.

> 
> 
> P.S. I've noticed an interesting detail after submitting the patch. Produced erofs image reports an enormous file size after conversion error:
> 
> $ mkfs.erofs --tar=f --aufs --quiet -Enoinline_data test.erofs test-no-end.tar
> -rw-r--r--    1 user  admin  2199023255552 Sep 29 13:43 test.erofs

Yes, that is expected according to the current codebase
since it uses a sparse file to keep temproary data and
truncate when successful.

I guess we should remove the image entirely if mkfs
fails instead.

Thanks,
Gao Xiang

> $ du -sh test.erofs
> 4,0K    test.erofs
> 

Re: [PATCH] erofs-utils: tar: support archives without end-of-archive entry

[Gao Xiang]

On 2025/9/30 16:01, Gao Xiang wrote:
> 
> 
> On 2025/9/30 15:42, Ivan Mikheykin wrote:
>> Hello!
>>
>> On 9/29/25 5:50 PM, Gao Xiang wrote:
>>
>>> Could you confirm how docker/containerd or podman parses such image?
>>
>> I confirm that images with such layers work well at least in containerd with overlayfs. We encounter problems during experiments with containerd and erofs. Also, GNU tar and BSD tar are good without end-of-archive zeros.
>>
>>>
>>> Because the POSIX standard says:
>>> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html
>>> "At the end of the archive file there shall be two 512-byte blocks filled
>>> with binary zeros, interpreted as an end-of-archive indicator."
>>>
>>> So such tar layers will be non-standard, I wonder we need at least
>>> a erofs_warn() message for such tars at least.
>>>
>>
>> Good point. I think I can add erofs_warn() message to the patch.
> 
> Yes, please.
> 
>>
>>
>> P.S. I've noticed an interesting detail after submitting the patch. Produced erofs image reports an enormous file size after conversion error:
>>
>> $ mkfs.erofs --tar=f --aufs --quiet -Enoinline_data test.erofs test-no-end.tar
>> -rw-r--r--    1 user  admin  2199023255552 Sep 29 13:43 test.erofs
> 
> Yes, that is expected according to the current codebase
> since it uses a sparse file to keep temproary data and
> truncate when successful.
> 
> I guess we should remove the image entirely if mkfs
> fails instead.

BTW, also I suggest you use `--sort=none` to get rid of
this extra copy, see:
https://github.com/containerd/containerd/blob/main/docs/snapshotters/erofs.md

> 
> Thanks,
> Gao Xiang
> 
>> $ du -sh test.erofs
>> 4,0K    test.erofs
>>
> 

[PATCH v2] erofs-utils: tar: support archives without end-of-archive entry

[Ivan Mikheykin]

Tar standard https://www.gnu.org/software/tar/manual/html_node/Standard.html
says that archive "terminated by an end-of-archive entry,
which consists of two 512 blocks of zero bytes".

Is also says:

"A reasonable system should write such end-of-file marker at the end
of an archive, but must not assume that such a block exists when
reading an archive. In particular, GNU tar does not treat missing
end-of-file marker as an error and silently ignores the fact."

It is rare for erofs to encounter such problem, as images are mostly
built with docker or buildah. But if you create image using tar library
in Golang directly uploading layers to registry, you'll get tar layers
without end-of-archive block. Running containers with such images will
trigger this error during extraction:

mkfs.erofs --tar=f --aufs --quiet -Enoinline_data test.erofs test-no-end.tar
<E> erofs: failed to read header block @ 42496
<E> erofs: 	Could not format the device : [Error 5] Input/output error

This patch fixes the problem by assuming that eof is equal to the end-of-archive.

Reproducible tar without end-of-archive (base64-encoded gzipped blob):
H4sICKVi2mgAA3Rlc3QtMTAtMi1ibG9ja3MudGFyAAtzDQr29PdjoCUwAAIzExMwbW5mCqYN
jQzANBgYGTEYmhqYmpqamRoaGTMYGBqaGJkyKBjQ1FVQUFpcklikoMCQkpmYll9ahFNdYkpu
Zh49HERfYKhnoWdowGVkYGSqa2Cua2jKNdAuGgX0BADwFwqsAAQAAA==

Also, add warning about non-conformant tar layers:

mkfs.erofs --tar=f --aufs -Enoinline_data test.erofs test-no-end.tar
mkfs.erofs 1.8.10-g0a2bc574-dirty
<W> erofs: unexpected end of file @ 1024 (may be non-standard tar without end of archive zeros)
Build completed.
...

Signed-off-by: Ivan Mikheykin <ivan.mikheykin@flant.com>
---
 lib/tar.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/tar.c b/lib/tar.c
index 100efb3..46a9c92 100644
--- a/lib/tar.c
+++ b/lib/tar.c
@@ -725,6 +725,11 @@ restart:
 	tar_offset = tar->offset;
 	ret = erofs_iostream_read(&tar->ios, (void **)&th, sizeof(*th));
 	if (ret != sizeof(*th)) {
+		if (tar->ios.feof) {
+			erofs_warn("unexpected end of file @ %llu (may be non-standard tar without end of archive zeros)", tar_offset);
+			ret = 1;
+			goto out;
+		}
 		if (tar->headeronly_mode || tar->ddtaridx_mode) {
 			ret = 1;
 			goto out;
-- 
2.39.3 (Apple Git-146)

Re: [PATCH v2] erofs-utils: tar: support archives without end-of-archive entry

[Gao Xiang]

On 2025/10/2 01:13, Ivan Mikheykin wrote:
> Tar standard https://www.gnu.org/software/tar/manual/html_node/Standard.html
> says that archive "terminated by an end-of-archive entry,
> which consists of two 512 blocks of zero bytes".
> 
> Is also says:
> 
> "A reasonable system should write such end-of-file marker at the end
> of an archive, but must not assume that such a block exists when
> reading an archive. In particular, GNU tar does not treat missing
> end-of-file marker as an error and silently ignores the fact."
> 
> It is rare for erofs to encounter such problem, as images are mostly
> built with docker or buildah. But if you create image using tar library
> in Golang directly uploading layers to registry, you'll get tar layers
> without end-of-archive block. Running containers with such images will
> trigger this error during extraction:
> 
> mkfs.erofs --tar=f --aufs --quiet -Enoinline_data test.erofs test-no-end.tar
> <E> erofs: failed to read header block @ 42496
> <E> erofs: 	Could not format the device : [Error 5] Input/output error
> 
> This patch fixes the problem by assuming that eof is equal to the end-of-archive.
> 
> Reproducible tar without end-of-archive (base64-encoded gzipped blob):
> H4sICKVi2mgAA3Rlc3QtMTAtMi1ibG9ja3MudGFyAAtzDQr29PdjoCUwAAIzExMwbW5mCqYN
> jQzANBgYGTEYmhqYmpqamRoaGTMYGBqaGJkyKBjQ1FVQUFpcklikoMCQkpmYll9ahFNdYkpu
> Zh49HERfYKhnoWdowGVkYGSqa2Cua2jKNdAuGgX0BADwFwqsAAQAAA==
> 
> Also, add warning about non-conformant tar layers:
> 
> mkfs.erofs --tar=f --aufs -Enoinline_data test.erofs test-no-end.tar
> mkfs.erofs 1.8.10-g0a2bc574-dirty
> <W> erofs: unexpected end of file @ 1024 (may be non-standard tar without end of archive zeros)
> Build completed.
> ...
> 
> Signed-off-by: Ivan Mikheykin <ivan.mikheykin@flant.com>

Thanks, will apply.

Thanks,
Gao Xiang