[PATCH] archive: make --add-virtual-file honor --prefix

[PATCH] archive: make --add-virtual-file honor --prefix

[Tom Scogland via GitGitGadget]

From: Tom Scogland <scogland1@llnl.gov>

The documentation for archive states:

  The path of the file in the archive is built by concatenating the
  value of the last `--prefix` moption (if any) before this
  `--add-virtual-file` and <path>.

This matches the documentation for --add-file and the behavior works for
that option, but --prefix is ignored for --add-virtual-file.

This commit modifies archive.c to include the prefix in the path and
adds a check into the existing add-virtual-file test to ensure that it
honors both the most recent prefix before the flag.

In looking for others with this issue, I found message
a143e25a70b44b82b4ee6fa3bb2bcda4@atlas-elektronik.com on the mailing
list, where Stefan proposed a basically identical patch to archive.c
back in February, so the main addition here is the test along with that
patch.

Signed-off-by: Tom Scogland <scogland1@llnl.gov>
---
    archive: make --add-virtual-file honor --prefix

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1719%2Ftrws%2Fhonor-prefix-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1719/trws/honor-prefix-v1
Pull-Request: https://github.com/git/git/pull/1719

 archive.c              | 11 +++++------
 t/t5003-archive-zip.sh |  8 ++++++--
 2 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/archive.c b/archive.c
index 5287fcdd8e0..90086a1708f 100644
--- a/archive.c
+++ b/archive.c
@@ -365,12 +365,11 @@ int write_archive_entries(struct archiver_args *args,
 
 		put_be64(fake_oid.hash, i + 1);
 
+		strbuf_reset(&path_in_archive);
+		if (info->base)
+			strbuf_addstr(&path_in_archive, info->base);
+		strbuf_addstr(&path_in_archive, basename(path));
 		if (!info->content) {
-			strbuf_reset(&path_in_archive);
-			if (info->base)
-				strbuf_addstr(&path_in_archive, info->base);
-			strbuf_addstr(&path_in_archive, basename(path));
-
 			strbuf_reset(&content);
 			if (strbuf_read_file(&content, path, info->stat.st_size) < 0)
 				err = error_errno(_("cannot read '%s'"), path);
@@ -381,7 +380,7 @@ int write_archive_entries(struct archiver_args *args,
 						  content.buf, content.len);
 		} else {
 			err = write_entry(args, &fake_oid,
-					  path, strlen(path),
+					  path_in_archive.buf, path_in_archive.len,
 					  canon_mode(info->stat.st_mode),
 					  info->content, info->stat.st_size);
 		}
diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh
index 961c6aac256..acc8bc4fcd6 100755
--- a/t/t5003-archive-zip.sh
+++ b/t/t5003-archive-zip.sh
@@ -218,14 +218,18 @@ test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' '
 	fi &&
 	git archive --format=zip >with_file_with_content.zip \
 		--add-virtual-file=\""$PATHNAME"\": \
-		--add-virtual-file=hello:world $EMPTY_TREE &&
+		--add-virtual-file=hello:world \
+		--prefix=subdir/ --add-virtual-file=hello:world \
+		--prefix= $EMPTY_TREE &&
 	test_when_finished "rm -rf tmp-unpack" &&
 	mkdir tmp-unpack && (
 		cd tmp-unpack &&
 		"$GIT_UNZIP" ../with_file_with_content.zip &&
 		test_path_is_file hello &&
 		test_path_is_file "$PATHNAME" &&
-		test world = $(cat hello)
+		test world = $(cat hello) &&
+		test_path_is_file subdir/hello &&
+		test world = $(cat subdir/hello)
 	)
 '
 

base-commit: 786a3e4b8d754d2b14b1208b98eeb0a554ef19a8
-- 
gitgitgadget

Re: [PATCH] archive: make --add-virtual-file honor --prefix

[Junio C Hamano]

"Tom Scogland via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Tom Scogland <scogland1@llnl.gov>
>
> The documentation for archive states:
>
>   The path of the file in the archive is built by concatenating the
>   value of the last `--prefix` moption (if any) before this
>   `--add-virtual-file` and <path>.
>
> This matches the documentation for --add-file and the behavior works for
> that option, but --prefix is ignored for --add-virtual-file.

This first paragraph was a bit hard to parse for me.

You will contrast the quoted paragraph with another one you did not
quote later in this paragraph, so it is more helpful to readers if
you instead said:

    The documentatation for archive describes its
    "--add-virtual-file" option like so:

        ... excerpt from --add-virtual-file description ...

    This description is the same as "--add-file", and "--add-file"
    does behave the way as described.  "--add-virtual-file" however
    ignores "--prefix".

> This commit modifies archive.c to include the prefix in the path and
> adds a check into the existing add-virtual-file test to ensure that it
> honors both the most recent prefix before the flag.

Style: "This comit modifies" -> "Modify".

An obvious alternative fix is to update the documentation, which
would be a much safer thing to do, given that there may be existing
scripts written during the two years since --add-virtual-file option
was introduced and has been behaving exactly this way.  They will
all be broken big time once the command starts honoring the
"--prefix" option.

> In looking for others with this issue, I found message
> a143e25a70b44b82b4ee6fa3bb2bcda4@atlas-elektronik.com on the mailing
> list, where Stefan proposed a basically identical patch to archive.c
> back in February, so the main addition here is the test along with that
> patch.

This pargraph should come _after_ the three-dash lines below.

> Signed-off-by: Tom Scogland <scogland1@llnl.gov>
> ---
>     archive: make --add-virtual-file honor --prefix

The implementation looked obvious, assuming that it is a good idea
to change it (I've already talked about a safer alternative fix).

> diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh
> index 961c6aac256..acc8bc4fcd6 100755
> --- a/t/t5003-archive-zip.sh
> +++ b/t/t5003-archive-zip.sh
> @@ -218,14 +218,18 @@ test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' '
>  	fi &&
>  	git archive --format=zip >with_file_with_content.zip \
>  		--add-virtual-file=\""$PATHNAME"\": \
> -		--add-virtual-file=hello:world $EMPTY_TREE &&
> +		--add-virtual-file=hello:world \
> +		--prefix=subdir/ --add-virtual-file=hello:world \
> +		--prefix= $EMPTY_TREE &&

Instead of reusing the exactly the same name and contents, use
something different so that it is clear to the later test which of
the two "--add-virtual-file" options created these two paths in the
unpacked directories.  I.e., create something like

	--prefix=subdir/ --add-virtual-file=good:night

here and update the test below to match.

>  	test_when_finished "rm -rf tmp-unpack" &&
>  	mkdir tmp-unpack && (
>  		cd tmp-unpack &&
>  		"$GIT_UNZIP" ../with_file_with_content.zip &&
>  		test_path_is_file hello &&
>  		test_path_is_file "$PATHNAME" &&
> -		test world = $(cat hello)
> +		test world = $(cat hello) &&
> +		test_path_is_file subdir/hello &&
> +		test world = $(cat subdir/hello)
>  	)
>  '

Other than that, looks good to me.  Thanks.

Re: [PATCH] archive: make --add-virtual-file honor --prefix

[Tom Scogland]

On 15 May 2024, at 8:23, Junio C Hamano wrote:

> "Tom Scogland via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
>> From: Tom Scogland <scogland1@llnl.gov>
>>
>> The documentation for archive states:
>>
>>   The path of the file in the archive is built by concatenating the
>>   value of the last `--prefix` moption (if any) before this
>>   `--add-virtual-file` and <path>.
>>
>> This matches the documentation for --add-file and the behavior works for
>> that option, but --prefix is ignored for --add-virtual-file.
>
> This first paragraph was a bit hard to parse for me.
>
> You will contrast the quoted paragraph with another one you did not
> quote later in this paragraph, so it is more helpful to readers if
> you instead said:
>
>     The documentatation for archive describes its
>     "--add-virtual-file" option like so:
>
>         ... excerpt from --add-virtual-file description ...
>
>     This description is the same as "--add-file", and "--add-file"
>     does behave the way as described.  "--add-virtual-file" however
>     ignores "--prefix".

Ok, I'll update the message with this and the below style.

>> This commit modifies archive.c to include the prefix in the path and
>> adds a check into the existing add-virtual-file test to ensure that it
>> honors both the most recent prefix before the flag.
>
> Style: "This comit modifies" -> "Modify".
>
> An obvious alternative fix is to update the documentation, which
> would be a much safer thing to do, given that there may be existing
> scripts written during the two years since --add-virtual-file option
> was introduced and has been behaving exactly this way.  They will
> all be broken big time once the command starts honoring the
> "--prefix" option.

I wouldn't mind doing this necessarily, but would want an option that follows the documentation in addition.  The current implementation is harder to use consistently with `--add-file` and with `--prefix`, though it's possible to manually prefix each virtual file it's surprising that it produces (as far as I have found) the only files in the archive that don't fall under the prefix.

>> In looking for others with this issue, I found message
>> a143e25a70b44b82b4ee6fa3bb2bcda4@atlas-elektronik.com on the mailing
>> list, where Stefan proposed a basically identical patch to archive.c
>> back in February, so the main addition here is the test along with that
>> patch.
>
> This pargraph should come _after_ the three-dash lines below.

Certainly.

>> Signed-off-by: Tom Scogland <scogland1@llnl.gov>
>> ---
>>     archive: make --add-virtual-file honor --prefix
>
> The implementation looked obvious, assuming that it is a good idea
> to change it (I've already talked about a safer alternative fix).
>
>> diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh
>> index 961c6aac256..acc8bc4fcd6 100755
>> --- a/t/t5003-archive-zip.sh
>> +++ b/t/t5003-archive-zip.sh
>> @@ -218,14 +218,18 @@ test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' '
>>  	fi &&
>>  	git archive --format=zip >with_file_with_content.zip \
>>  		--add-virtual-file=\""$PATHNAME"\": \
>> -		--add-virtual-file=hello:world $EMPTY_TREE &&
>> +		--add-virtual-file=hello:world \
>> +		--prefix=subdir/ --add-virtual-file=hello:world \
>> +		--prefix= $EMPTY_TREE &&
>
> Instead of reusing the exactly the same name and contents, use
> something different so that it is clear to the later test which of
> the two "--add-virtual-file" options created these two paths in the
> unpacked directories.  I.e., create something like
>
>     --prefix=subdir/ --add-virtual-file=good:night
>
> here and update the test below to match.
>
>>  	test_when_finished "rm -rf tmp-unpack" &&
>>  	mkdir tmp-unpack && (
>>  		cd tmp-unpack &&
>>  		"$GIT_UNZIP" ../with_file_with_content.zip &&
>>  		test_path_is_file hello &&
>>  		test_path_is_file "$PATHNAME" &&
>> -		test world = $(cat hello)
>> +		test world = $(cat hello) &&
>> +		test_path_is_file subdir/hello &&
>> +		test world = $(cat subdir/hello)
>>  	)
>>  '
>
> Other than that, looks good to me.  Thanks.

Thanks for the feedback, I'll get an updated patch posted later today.

[PATCH v2] archive: make --add-virtual-file honor --prefix

[Tom Scogland via GitGitGadget]

From: Tom Scogland <scogland1@llnl.gov>

The documentation for archive describes the `--add-virtual-file` option
thusly:

  The path of the file in the archive is built by concatenating the
  value of the last `--prefix` moption (if any) before this
  `--add-virtual-file` and <path>.

The `--add-file` documentation is similar:

  The path of the file in the archive is built by concatenating the
  value of the last --prefix option (if any) before this --add-file and
  the basename of <file>.

Notably both explicitly state that they honor the last `--prefix` option
before the `--add` option in question.  The implementation of
`--add-file` seems to have always honored prefix, but the implementation
of `--add-virtual-file` does not.  Also note that `--add-virtual-file`
explicitly states it will use the full path given, while `--add-file`
uses the basename of the path it is given.

Modify archive.c to include the prefix in the path used by
`--add-virtual-file` and add checks into
the existing add-virtual-file test to verify:

* that `--prefix` is honored
* that leading path components are preserved
* that both work together and separately

Changes since v1:
- Revised the commit message style
- Added tests for basename/non-basename behavior
- Fixed archive.c to use full path for virtual and basename for add-file

Signed-off-by: Tom Scogland <scogland1@llnl.gov>
---
    archive: make --add-virtual-file honor --prefix

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1719%2Ftrws%2Fhonor-prefix-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1719/trws/honor-prefix-v2
Pull-Request: https://github.com/git/git/pull/1719

Range-diff vs v1:

 1:  1b685d8ca1a ! 1:  b9a0ef282a3 archive: make --add-virtual-file honor --prefix
     @@ Metadata
       ## Commit message ##
          archive: make --add-virtual-file honor --prefix
      
     -    The documentation for archive states:
     +    The documentation for archive describes the `--add-virtual-file` option
     +    thusly:
      
            The path of the file in the archive is built by concatenating the
            value of the last `--prefix` moption (if any) before this
            `--add-virtual-file` and <path>.
      
     -    This matches the documentation for --add-file and the behavior works for
     -    that option, but --prefix is ignored for --add-virtual-file.
     +    The `--add-file` documentation is similar:
      
     -    This commit modifies archive.c to include the prefix in the path and
     -    adds a check into the existing add-virtual-file test to ensure that it
     -    honors both the most recent prefix before the flag.
     +      The path of the file in the archive is built by concatenating the
     +      value of the last --prefix option (if any) before this --add-file and
     +      the basename of <file>.
     +
     +    Notably both explicitly state that they honor the last `--prefix` option
     +    before the `--add` option in question.  The implementation of
     +    `--add-file` seems to have always honored prefix, but the implementation
     +    of `--add-virtual-file` does not.  Also note that `--add-virtual-file`
     +    explicitly states it will use the full path given, while `--add-file`
     +    uses the basename of the path it is given.
     +
     +    Modify archive.c to include the prefix in the path used by
     +    `--add-virtual-file` and add checks into
     +    the existing add-virtual-file test to verify:
     +
     +    * that `--prefix` is honored
     +    * that leading path components are preserved
     +    * that both work together and separately
      
     -    In looking for others with this issue, I found message
     -    a143e25a70b44b82b4ee6fa3bb2bcda4@atlas-elektronik.com on the mailing
     -    list, where Stefan proposed a basically identical patch to archive.c
     -    back in February, so the main addition here is the test along with that
     -    patch.
     +    Changes since v1:
     +    - Revised the commit message style
     +    - Added tests for basename/non-basename behavior
     +    - Fixed archive.c to use full path for virtual and basename for add-file
      
          Signed-off-by: Tom Scogland <scogland1@llnl.gov>
      
     @@ archive.c: int write_archive_entries(struct archiver_args *args,
      +		strbuf_reset(&path_in_archive);
      +		if (info->base)
      +			strbuf_addstr(&path_in_archive, info->base);
     -+		strbuf_addstr(&path_in_archive, basename(path));
       		if (!info->content) {
      -			strbuf_reset(&path_in_archive);
      -			if (info->base)
      -				strbuf_addstr(&path_in_archive, info->base);
     --			strbuf_addstr(&path_in_archive, basename(path));
     + 			strbuf_addstr(&path_in_archive, basename(path));
      -
       			strbuf_reset(&content);
       			if (strbuf_read_file(&content, path, info->stat.st_size) < 0)
       				err = error_errno(_("cannot read '%s'"), path);
      @@ archive.c: int write_archive_entries(struct archiver_args *args,
     + 						  canon_mode(info->stat.st_mode),
       						  content.buf, content.len);
       		} else {
     ++			strbuf_addstr(&path_in_archive, path);
       			err = write_entry(args, &fake_oid,
      -					  path, strlen(path),
      +					  path_in_archive.buf, path_in_archive.len,
     @@ t/t5003-archive-zip.sh: test_expect_success UNZIP 'git archive --format=zip --ad
       		--add-virtual-file=\""$PATHNAME"\": \
      -		--add-virtual-file=hello:world $EMPTY_TREE &&
      +		--add-virtual-file=hello:world \
     -+		--prefix=subdir/ --add-virtual-file=hello:world \
     ++		--add-virtual-file=with/dir/noprefix:withdirnopre \
     ++		--prefix=subdir/ --add-virtual-file=with/dirprefix:withdirprefix \
     ++		--prefix=subdir2/ --add-virtual-file=withoutdir:withoutdir \
      +		--prefix= $EMPTY_TREE &&
       	test_when_finished "rm -rf tmp-unpack" &&
       	mkdir tmp-unpack && (
     @@ t/t5003-archive-zip.sh: test_expect_success UNZIP 'git archive --format=zip --ad
       		test_path_is_file "$PATHNAME" &&
      -		test world = $(cat hello)
      +		test world = $(cat hello) &&
     -+		test_path_is_file subdir/hello &&
     -+		test world = $(cat subdir/hello)
     ++		test_path_is_file with/dir/noprefix &&
     ++		test withdirnopre = $(cat with/dir/noprefix) &&
     ++		test_path_is_file subdir/with/dirprefix &&
     ++		test withdirprefix = $(cat subdir/with/dirprefix) &&
     ++		test_path_is_file subdir2/withoutdir &&
     ++		test withoutdir = $(cat subdir2/withoutdir)
       	)
       '
       


 archive.c              | 10 +++++-----
 t/t5003-archive-zip.sh | 14 ++++++++++++--
 2 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/archive.c b/archive.c
index 5287fcdd8e0..64777a9870d 100644
--- a/archive.c
+++ b/archive.c
@@ -365,12 +365,11 @@ int write_archive_entries(struct archiver_args *args,
 
 		put_be64(fake_oid.hash, i + 1);
 
+		strbuf_reset(&path_in_archive);
+		if (info->base)
+			strbuf_addstr(&path_in_archive, info->base);
 		if (!info->content) {
-			strbuf_reset(&path_in_archive);
-			if (info->base)
-				strbuf_addstr(&path_in_archive, info->base);
 			strbuf_addstr(&path_in_archive, basename(path));
-
 			strbuf_reset(&content);
 			if (strbuf_read_file(&content, path, info->stat.st_size) < 0)
 				err = error_errno(_("cannot read '%s'"), path);
@@ -380,8 +379,9 @@ int write_archive_entries(struct archiver_args *args,
 						  canon_mode(info->stat.st_mode),
 						  content.buf, content.len);
 		} else {
+			strbuf_addstr(&path_in_archive, path);
 			err = write_entry(args, &fake_oid,
-					  path, strlen(path),
+					  path_in_archive.buf, path_in_archive.len,
 					  canon_mode(info->stat.st_mode),
 					  info->content, info->stat.st_size);
 		}
diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh
index 961c6aac256..0cf3aef8ace 100755
--- a/t/t5003-archive-zip.sh
+++ b/t/t5003-archive-zip.sh
@@ -218,14 +218,24 @@ test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' '
 	fi &&
 	git archive --format=zip >with_file_with_content.zip \
 		--add-virtual-file=\""$PATHNAME"\": \
-		--add-virtual-file=hello:world $EMPTY_TREE &&
+		--add-virtual-file=hello:world \
+		--add-virtual-file=with/dir/noprefix:withdirnopre \
+		--prefix=subdir/ --add-virtual-file=with/dirprefix:withdirprefix \
+		--prefix=subdir2/ --add-virtual-file=withoutdir:withoutdir \
+		--prefix= $EMPTY_TREE &&
 	test_when_finished "rm -rf tmp-unpack" &&
 	mkdir tmp-unpack && (
 		cd tmp-unpack &&
 		"$GIT_UNZIP" ../with_file_with_content.zip &&
 		test_path_is_file hello &&
 		test_path_is_file "$PATHNAME" &&
-		test world = $(cat hello)
+		test world = $(cat hello) &&
+		test_path_is_file with/dir/noprefix &&
+		test withdirnopre = $(cat with/dir/noprefix) &&
+		test_path_is_file subdir/with/dirprefix &&
+		test withdirprefix = $(cat subdir/with/dirprefix) &&
+		test_path_is_file subdir2/withoutdir &&
+		test withoutdir = $(cat subdir2/withoutdir)
 	)
 '
 

base-commit: 786a3e4b8d754d2b14b1208b98eeb0a554ef19a8
-- 
gitgitgadget

Re: [PATCH v2] archive: make --add-virtual-file honor --prefix

[Junio C Hamano]

"Tom Scogland via GitGitGadget" <gitgitgadget@gmail.com> writes:

> Notably both explicitly state that they honor the last `--prefix` option
> before the `--add` option in question.  The implementation of
> `--add-file` seems to have always honored prefix, but the implementation
> of `--add-virtual-file` does not.

The above is misleading.

    The implementation of `--add-file` has always honored the prefix,
    while the implementation of `--add-virtual-file` has always ignored
    the prefix.

would make it easier to assess how long existing users may have been
relying on the current behaviour.

> Also note that `--add-virtual-file`
> explicitly states it will use the full path given, while `--add-file`
> uses the basename of the path it is given.

Yes, this is a very good thing to mention.  It is probably the
reason why the implementation decided not to add prefix to the "full
path" that already can have the leading directories.

> Modify archive.c to include the prefix in the path used by
> `--add-virtual-file` and add checks into
> the existing add-virtual-file test to verify:
>
> * that `--prefix` is honored
> * that leading path components are preserved
> * that both work together and separately

Very nice job explaining the chosen design clearly (even though I do
not necessarily agree with the direction this patch is going).

Also, given that this option was introduced for an explicit purpose
of using it to write out diagnostics archive file, we should mention
that this change does not break it in the proposed log message, at
least.  Of course, we should do so after verifying that is indeed
the case, and better yet, after verifying that it will be hard for
future changes to diagnose.c to trigger an unexpected behaviour
caused by this change [*].

> Changes since v1:
> - Revised the commit message style
> - Added tests for basename/non-basename behavior
> - Fixed archive.c to use full path for virtual and basename for add-file

The "changes since v1" section does not belong to the log message
proper, as v1 never happened as long as readers of "git log" are
concerned.  It is a very good thing to help reviewers to have below
the three-dash lines that comes after your sign-off, though.

> Signed-off-by: Tom Scogland <scogland1@llnl.gov>
> ---

>  archive.c              | 10 +++++-----
>  t/t5003-archive-zip.sh | 14 ++++++++++++--
>  2 files changed, 17 insertions(+), 7 deletions(-)
>
> diff --git a/archive.c b/archive.c
> index 5287fcdd8e0..64777a9870d 100644
> --- a/archive.c
> +++ b/archive.c
> @@ -365,12 +365,11 @@ int write_archive_entries(struct archiver_args *args,
>  
>  		put_be64(fake_oid.hash, i + 1);
>  
> +		strbuf_reset(&path_in_archive);
> +		if (info->base)
> +			strbuf_addstr(&path_in_archive, info->base);
>  		if (!info->content) {
> -			strbuf_reset(&path_in_archive);
> -			if (info->base)
> -				strbuf_addstr(&path_in_archive, info->base);
>  			strbuf_addstr(&path_in_archive, basename(path));
> -
>  			strbuf_reset(&content);
>  			if (strbuf_read_file(&content, path, info->stat.st_size) < 0)
>  				err = error_errno(_("cannot read '%s'"), path);
> @@ -380,8 +379,9 @@ int write_archive_entries(struct archiver_args *args,
>  						  canon_mode(info->stat.st_mode),
>  						  content.buf, content.len);
>  		} else {
> +			strbuf_addstr(&path_in_archive, path);
>  			err = write_entry(args, &fake_oid,
> -					  path, strlen(path),
> +					  path_in_archive.buf, path_in_archive.len,
>  					  canon_mode(info->stat.st_mode),
>  					  info->content, info->stat.st_size);
>  		}
> diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh
> index 961c6aac256..0cf3aef8ace 100755
> --- a/t/t5003-archive-zip.sh
> +++ b/t/t5003-archive-zip.sh
> @@ -218,14 +218,24 @@ test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' '
>  	fi &&
>  	git archive --format=zip >with_file_with_content.zip \
>  		--add-virtual-file=\""$PATHNAME"\": \
> -		--add-virtual-file=hello:world $EMPTY_TREE &&
> +		--add-virtual-file=hello:world \
> +		--add-virtual-file=with/dir/noprefix:withdirnopre \
> +		--prefix=subdir/ --add-virtual-file=with/dirprefix:withdirprefix \
> +		--prefix=subdir2/ --add-virtual-file=withoutdir:withoutdir \
> +		--prefix= $EMPTY_TREE &&
>  	test_when_finished "rm -rf tmp-unpack" &&
>  	mkdir tmp-unpack && (
>  		cd tmp-unpack &&
>  		"$GIT_UNZIP" ../with_file_with_content.zip &&
>  		test_path_is_file hello &&
>  		test_path_is_file "$PATHNAME" &&
> -		test world = $(cat hello)
> +		test world = $(cat hello) &&
> +		test_path_is_file with/dir/noprefix &&
> +		test withdirnopre = $(cat with/dir/noprefix) &&
> +		test_path_is_file subdir/with/dirprefix &&
> +		test withdirprefix = $(cat subdir/with/dirprefix) &&
> +		test_path_is_file subdir2/withoutdir &&
> +		test withoutdir = $(cat subdir2/withoutdir)

OK.  With different payload at different paths, it is easier than
the previous round to see where things are expected to go in the
result.


[Footnote]

 * I got curious and did this part for you.  After calling
   add_directory_to_archiver() that uses "--prefix" to move the
   target directory around in the output hierarchy, the code clears
   with "--prefix="---even a future change to diagnose.c adds more
   uses to --add-virtual-file= after it happens, it will not go to
   deep in the directory hierarchy where the last use of "--prefix"
   happened to be pointing at.

Re: [PATCH v2] archive: make --add-virtual-file honor --prefix

[Tom Scogland]

On 17 May 2024, at 16:33, Junio C Hamano wrote:

> "Tom Scogland via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
>> Notably both explicitly state that they honor the last `--prefix` option
>> before the `--add` option in question.  The implementation of
>> `--add-file` seems to have always honored prefix, but the implementation
>> of `--add-virtual-file` does not.
>
> The above is misleading.
>
>     The implementation of `--add-file` has always honored the prefix,
>     while the implementation of `--add-virtual-file` has always ignored
>     the prefix.
>
> would make it easier to assess how long existing users may have been
> relying on the current behaviour.

Fair, I had no intention to mislead and will reword.

>> Modify archive.c to include the prefix in the path used by
>> `--add-virtual-file` and add checks into
>> the existing add-virtual-file test to verify:
>>
>> * that `--prefix` is honored
>> * that leading path components are preserved
>> * that both work together and separately
>
> Very nice job explaining the chosen design clearly (even though I do
> not necessarily agree with the direction this patch is going).

Thanks for that.  As to the direction, I mentioned earlier adding a different flag, or perhaps marking the filename in some fashion to express that the prefix should be honored, would you prefer that? It would, as you said, be much safer in that there's no reason for it to be a breaking change. If there's a design you prefer that would result in having an opt-in way to get the prefix behavior I wouldn't mind implementing it.

> Also, given that this option was introduced for an explicit purpose
> of using it to write out diagnostics archive file, we should mention
> that this change does not break it in the proposed log message, at
> least.  Of course, we should do so after verifying that is indeed
> the case, and better yet, after verifying that it will be hard for
> future changes to diagnose.c to trigger an unexpected behaviour
> caused by this change [*].

That's a very good point, and thank you for digging into it.

>> Changes since v1:
>> - Revised the commit message style
>> - Added tests for basename/non-basename behavior
>> - Fixed archive.c to use full path for virtual and basename for add-file
>
> The "changes since v1" section does not belong to the log message
> proper, as v1 never happened as long as readers of "git log" are
> concerned.  It is a very good thing to help reviewers to have below
> the three-dash lines that comes after your sign-off, though.

My apologies, this is my unfamiliarity with GitGitGadget, I'll put information like this in the PR description next time, which I think will do that.

Re: [PATCH v2] archive: make --add-virtual-file honor --prefix

[René Scharfe]

Am 17.05.24 um 19:34 schrieb Tom Scogland via GitGitGadget:
> From: Tom Scogland <scogland1@llnl.gov>
>
> The documentation for archive describes the `--add-virtual-file` option
> thusly:
>
>   The path of the file in the archive is built by concatenating the
>   value of the last `--prefix` moption (if any) before this
>   `--add-virtual-file` and <path>.

The documentation does not actually misspell "option" as "moption".

> The `--add-file` documentation is similar:
>
>   The path of the file in the archive is built by concatenating the
>   value of the last --prefix option (if any) before this --add-file and
>   the basename of <file>.
>
> Notably both explicitly state that they honor the last `--prefix` option
> before the `--add` option in question.  The implementation of
> `--add-file` seems to have always honored prefix, but the implementation
> of `--add-virtual-file` does not.  Also note that `--add-virtual-file`
> explicitly states it will use the full path given, while `--add-file`
> uses the basename of the path it is given.
>
> Modify archive.c to include the prefix in the path used by
> `--add-virtual-file`

Aligning code and docs is a good idea.  Have you considered keeping the
code as is and changing the documentation instead, though?

The two options are related in that they both add untracked files, but
they necessarily have different arguments:

   --add-file=<file>
   --add-virtual-file=<path>:<content>

You can already specify any path you want with --add-virtual-file.
What's the advantage of honoring --prefix as well?

René

Re: [PATCH v2] archive: make --add-virtual-file honor --prefix

[Tom Scogland]

On 19 May 2024, at 6:25, René Scharfe wrote:

>
> Aligning code and docs is a good idea.  Have you considered keeping the
> code as is and changing the documentation instead, though?
>
> The two options are related in that they both add untracked files, but
> they necessarily have different arguments:
>
>    --add-file=<file>
>    --add-virtual-file=<path>:<content>
>
> You can already specify any path you want with --add-virtual-file.
> What's the advantage of honoring --prefix as well?

I came into this after trying to translate an --add-file to an --add-virtual-file and being surprised that the prefix wasn't applied. anything else you add, in the repo or not, it gets the prefix. I can go back and explicitly add the prefix, but it makes the options less naturally composable in my opinion.  The specific case was packaging some software that uses git metadata to set version and some other things when in a repo, and files in a tarball. The tarball's prefix is set in one part of the packaging code, files and exclusions in another, and passing that through was something we didn't have to think about using add-file, but do with add-virtual-file.

That said, it sounds like both you and Junio prefer updating the docs rather than the code, which makes me think I'm in the minority in that opinion.  If that's the case, I can certainly update the docs, and I imagine we can backport that easily wherever it makes sense.  I would really like to have the option to have the prefix apply though, either adding a new flag or an option to the existing one that would be invalid given current syntax or similar to provide the option.

Re: [PATCH v2] archive: make --add-virtual-file honor --prefix

[René Scharfe]

Am 20.05.24 um 18:10 schrieb Tom Scogland:
>
> On 19 May 2024, at 6:25, René Scharfe wrote:
>
>> You can already specify any path you want with --add-virtual-file.
>> What's the advantage of honoring --prefix as well?
>
> I came into this after trying to translate an --add-file to an
> --add-virtual-file and being surprised that the prefix wasn't
> applied.

Understandable, the documentation promised otherwise and the options
have very similar names.

> anything else you add, in the repo or not, it gets the prefix.> I can go back and explicitly add the prefix, but it makes the
> options less naturally composable in my opinion.

True, applying the prefix to all items would be simpler overall.

Speaking of simpler: --add-virtual-file could have been implemented to
only take a single argument -- the content -- and rely on --prefix to
provide the full path.  That's more consistent with other options, as
most of them only take single-valued arguments (or none). :]

> That said, it sounds like both you and Junio prefer updating the docs
> rather than the code, which makes me think I'm in the minority in
> that opinion.

I'm not sure I have an opinion on that topic, yet.  Fixing the
documentation is certainly easier.  Adding the prefix to the path of
virtual files as well is a breaking change.  I feel that the easier
route should at least be mentioned in the commit message and why it
was not taken.

> If that's the case, I can certainly update the docs, and I imagine we
> can backport that easily wherever it makes sense.  I would really
> like to have the option to have the prefix apply though, either
> adding a new flag or an option to the existing one that would be
> invalid given current syntax or similar to provide the option.
You mean like replacing a leading colon in the path with the prefix?

René

Re: [PATCH v2] archive: make --add-virtual-file honor --prefix

[Junio C Hamano]

Tom Scogland <scogland1@llnl.gov> writes:

> That said, it sounds like both you and Junio prefer updating the
> docs rather than the code, which makes me think I'm in the
> minority in that opinion.  If that's the case, I can certainly
> update the docs, and I imagine we can backport that easily
> wherever it makes sense.  I would really like to have the option
> to have the prefix apply though, either adding a new flag or an
> option to the existing one that would be invalid given current
> syntax or similar to provide the option.

[jc: wrapped overly long lines]

Wouldn't

    #!/bin/sh
    prefix=foo/bar
    git archive --prefix="$prefix" --add-file=x --add-file=y \
	--add-virtual-file="$prefix/path:contents"

be an option enough?  You only have to define $prefix once.

Re: [PATCH v2] archive: make --add-virtual-file honor --prefix

[Junio C Hamano]

René Scharfe <l.s.r@web.de> writes:

> I'm not sure I have an opinion on that topic, yet.  Fixing the
> documentation is certainly easier.  Adding the prefix to the path of
> virtual files as well is a breaking change.  I feel that the easier
> route should at least be mentioned in the commit message and why it
> was not taken.

It has been a few weeks since this discussion stalled.  Let me make
an executive decision on the direction here---let's keep the behaviour
and align the documentation so that we won't break existing users.

Thanks.

[PATCH] archive: document that --add-virtual-file takes full path

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> René Scharfe <l.s.r@web.de> writes:
>
>> I'm not sure I have an opinion on that topic, yet.  Fixing the
>> documentation is certainly easier.  Adding the prefix to the path of
>> virtual files as well is a breaking change.  I feel that the easier
>> route should at least be mentioned in the commit message and why it
>> was not taken.
>
> It has been a few weeks since this discussion stalled.  Let me make
> an executive decision on the direction here---let's keep the behaviour
> and align the documentation so that we won't break existing users.
>
> Thanks.

So here is to re-ignite the discussion.

----- >8 -----
Subject: [PATCH] archive: document that --add-virtual-file takes full path

Tom Scogland noticed that `--add-virtual-file` option uses the path
specified as its value as-is, without prepending any value given to
the `--prefix` option like `--add-file` does.

The behaviour has always been that way since the option was
introduced, but the documentation has always been wrong and said
that it would use the value of `--prefix` just like `--add-file`
does.

We could modify the behaviour to make it literally work like the
documentation said, but it would break existing scripts the users
use.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-archive.txt | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt
index 98526f2beb..a0e3fe7996 100644
--- a/Documentation/git-archive.txt
+++ b/Documentation/git-archive.txt
@@ -53,7 +53,7 @@ OPTIONS
 --prefix=<prefix>/::
 	Prepend <prefix>/ to paths in the archive.  Can be repeated; its
 	rightmost value is used for all tracked files.  See below which
-	value gets used by `--add-file` and `--add-virtual-file`.
+	value gets used by `--add-file`.
 
 -o <file>::
 --output=<file>::
@@ -67,9 +67,7 @@ OPTIONS
 
 --add-virtual-file=<path>:<content>::
 	Add the specified contents to the archive.  Can be repeated to add
-	multiple files.  The path of the file in the archive is built
-	by concatenating the value of the last `--prefix` option (if any)
-	before this `--add-virtual-file` and `<path>`.
+	multiple files.
 +
 The `<path>` argument can start and end with a literal double-quote
 character; the contained file name is interpreted as a C-style string,
@@ -81,6 +79,10 @@ if the path begins or ends with a double-quote character.
 The file mode is limited to a regular file, and the option may be
 subject to platform-dependent command-line limits. For non-trivial
 cases, write an untracked file and use `--add-file` instead.
++
+Note that unlike `--add-file` the path created in the archive is not
+affected by the `--prefix` option, as a full `<path>` can be given as
+the value of the option.
 
 --worktree-attributes::
 	Look for attributes in .gitattributes files in the working tree
-- 
2.45.2-683-g09b5b61c39

Re: [PATCH] archive: document that --add-virtual-file takes full path

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

>> It has been a few weeks since this discussion stalled.  Let me make
>> an executive decision on the direction here---let's keep the behaviour
>> and align the documentation so that we won't break existing users.
>>
>> Thanks.
>
> So here is to re-ignite the discussion.
>
> ----- >8 -----
> Subject: [PATCH] archive: document that --add-virtual-file takes full path

And it seems that nobody is interested in the topic all that much
anymore?

I've read the updated text once again, and didn't see anything
glaringly wrong, so I'll mark it for 'next'.

Thanks.

Re: [PATCH] archive: document that --add-virtual-file takes full path

[René Scharfe]

Am 26.06.24 um 00:43 schrieb Junio C Hamano:
> Junio C Hamano <gitster@pobox.com> writes:
>
>>> It has been a few weeks since this discussion stalled.  Let me make
>>> an executive decision on the direction here---let's keep the behaviour
>>> and align the documentation so that we won't break existing users.
>>>
>>> Thanks.
>>
>> So here is to re-ignite the discussion.
>>
>> ----- >8 -----
>> Subject: [PATCH] archive: document that --add-virtual-file takes full path
>
> And it seems that nobody is interested in the topic all that much
> anymore?

I'm low-key pleased to see a documentation bug being stomped.

Perhaps this saps the energy out of efforts to make --add-virtual-file
respect --prefix, but we'd probably need a new option name for that
anyway to keep backward compatibility (--add-prefix-file?), so the patch
adds no technical obstacle for that at least.

> I've read the updated text once again, and didn't see anything
> glaringly wrong, so I'll mark it for 'next'.

The patch looks good to me.  I like the new paragraph about --prefix not
being respected, in particular because I wouldn't thought of adding it,
but it does look helpful.

Perhaps add a Reported-by: line for Tom that's findable with trailer
tooling, though?

René