git support for "xattrs" (extended filesystem attributes)?

git support for "xattrs" (extended filesystem attributes)?

[Peter B.]

Hi everyone :)

I'm evaluating and testing extended attributes (xattrs) for professional 
archival collection use.
I couldn't find any proper documentation/information on how (well) git 
supports including xattrs in version control.

I assume behavior may be similar to any other property-change of 
files/folders?
Would it be possible to have bit-proof preservation of all xattrs? (even 
larger ones).

I'm grateful for any insights, pointers or comments. :D


Thanks in advance,
Peter B.

Re: git support for "xattrs" (extended filesystem attributes)?

[Junio C Hamano]

"Peter B." <pb@das-werkstatt.com> writes:

> I'm evaluating and testing extended attributes (xattrs) for
> professional archival collection use.
> I couldn't find any proper documentation/information on how (well) git
> supports including xattrs in version control.

Git tracks contents, pathnames where these contents are stored, and
the executable bit.  Nothing else.  Not even the owner/group of each
contents, or time of creation or the last modification.

I do not think Git is even aware of xattr at all.

Re: git support for "xattrs" (extended filesystem attributes)?

[Jeff King]

On Tue, Nov 26, 2024 at 08:40:32PM +0100, Peter B. wrote:

> I'm evaluating and testing extended attributes (xattrs) for professional
> archival collection use.
> I couldn't find any proper documentation/information on how (well) git
> supports including xattrs in version control.

As Junio noted, Git does not store most metadata at all (and most other
version control systems are the same).

There are some third party tools to save the information in a separate
file within the history and restore it on checkout. I haven't used them
myself and some of them are quite old, so YMMV:

  - https://etckeeper.branchable.com/

  - https://github.com/przemoc/metastore

-Peff

Re: git support for "xattrs" (extended filesystem attributes)?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> There are some third party tools to save the information in a separate
> file within the history and restore it on checkout.

Ah, you're right of course.  We are extensible that way.

> I haven't used them
> myself and some of them are quite old, so YMMV:
>
>   - https://etckeeper.branchable.com/
>
>   - https://github.com/przemoc/metastore

Re: git support for "xattrs" (extended filesystem attributes)?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2010 bytes --]

On 2024-11-26 at 19:40:32, Peter B. wrote:
> Hi everyone :)
> 
> I'm evaluating and testing extended attributes (xattrs) for professional
> archival collection use.
> I couldn't find any proper documentation/information on how (well) git
> supports including xattrs in version control.
> 
> I assume behavior may be similar to any other property-change of
> files/folders?
> Would it be possible to have bit-proof preservation of all xattrs? (even
> larger ones).

As others have mentioned, Git doesn't store this.  There are a couple
ways that one could store this information, depending on what you want
to store.

One way is to store the metadata in the `.gitattributes` file if you're
storing one or two well-known xattrs and the values are well stored as
text in the file.  You can then use `git ls-attr` and a `post-checkout`
hook[0] to set those attributes appropriately.

Another way is to store the data as part of an mtree(1) file in the
repository.  mtree is a file format that comes from the BSDs that can
contain a wide variety of data, including file type, uid and gid, text
user and group, permissions, size, mtime, cryptographic hashes, and a
wide variety of other data.  mtree utilities can usually generate a
manifest based on the state of the file system, validate that files
match a manifest, and usually update existing files such that they match
a manifest.  There's a Go-based version[1] which does support storing
xattrs.  Again, you can use a `post-checkout` hook here.

Since mtree is a key-value format, you can also extend the value with
other data.  I use a similar format in my dotfiles to specify install
location, for instance, so I recommend this as a good way to store this
data.

[0] Git LFS uses post-checkout hooks to adjust the read-only bit for
files which are lockable but not locked, so this is generally a robust
method.
[1] https://github.com/vbatts/go-mtree
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: git support for "xattrs" (extended filesystem attributes)?

[Peter B.]

Thank you everyone for your input!
This is very good now know.

I'll take a look at your suggestions, especially mtree and metastore, as 
they seem very useful in general.


Thanks and have a great day,
Peter

Re: git support for "xattrs" (extended filesystem attributes)?

[Peter B.]

On 11/27/24 02:29, brian m. carlson wrote:
> [...] There's a Go-based version[1] which does support storing
> xattrs.  Again, you can use a `post-checkout` hook here.
>
> Since mtree is a key-value format, you can also extend the value with
> other data.  I use a similar format in my dotfiles to specify install
> location, for instance, so I recommend this as a good way to store this
> data.

I just installed gomtree, and read my xattrs: looks very promising!
Thank you for the suggestion. :)

I'll read up on "post-checkout hooks" to see how I can integrate this in 
my git setup.


In case anyone's curious, here's an example of gomtree's output text for 
a file annotated using `yt-dlp --xattr`:

```
Universal\040Sequence\040\13320vPbH6UWIc].mp4 
xattr.user.xdg.referrer.url=aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj0yMHZQYkg2VVdJYw== 
xattr.user.dublincore.title=VW5pdmVyc2FsIFNlcXVlbmNl 
xattr.user.dublincore.date=MjAxNi0wOS0xNA== 
xattr.user.dublincore.contributor=Q29uc3BpcmFjeQ== 
xattr.user.dublincore.format=Mjk4IC0gMTI4MHg1NDAgKDcyMHA2MCkrMTQwIC0gYXVkaW8gb25seSAobWVkaXVtKQ== 
xattr.user.dublincore.description=VmlkZW8gcmVjb3JkaW5nIG9mIHRoZSBmaW5hbCB2ZXJzaW9uIG9mICJVbml2ZXJzYWwgU2VxdWVuY2UiLCBhIDY0IEtCeXRlIHJlYWx0aW1lIGRlbW8gYnkgQ29uc3BpcmFjeS4gRmlyc3QgcHJlc2VudGVkIGF0IHRoZSBGdW5jdGlvbiAyMDE2IGRlbW9wYXJ0eSwgd2hlcmUgaXQgcmVhY2hlZCAxc3QgcGxhY2UuCgpEb3dubG9hZCB0aGUgcmVhbHRpbWUgdmVyc2lvbiBhbmQgc291bmR0cmFjayBhdApodHRwOi8vY29uc3BpcmFjeS5odS9yZWxlYXNlLzY0ay91bml2ZXJzYWxfc2VxdWVuY2UvCgpFbmdpbmUgY29kZSBhbmQgZWZmZWN0cyBieQpCT1lDCgpWaXN1YWxzIGFuZCBkaXJlY3Rpb24gYnkKWk9PTQoKU3ludGggYW5kIG11c2ljIGJ5CkdBUkdBSgoKaHR0cDovL3d3dy5jb25zcGlyYWN5Lmh1Cmh0dHA6Ly9mYWNlYm9vay5jb20vY29uc3BpcmFjeS5odQpodHRwOi8vdHdpdHRlci5jb20vY29uc3BpcmFjeWh1CgojY29uc3BpcmFjeSAjZGVtb3NjZW5lICM2NGsgI2ludHJvICNyZWFsdGltZSAjaHVuZ2FyeQ==
```

Values are base64-encoded.
Nice.



Regards,
Peter B.