Understanding Index Header

Understanding Index Header

[Farhan Khan]

Hi all,

I am writing a program that parses out the .git/index file. I am
reading the git index header documentation, but I seem to be getting
jibberish data.
https://github.com/git/git/blob/master/Documentation/technical/index-format.txt

The first 12 bytes are the signature, version and entries. Great so far.

Afterwards, I try to read the extension signature, which the
documentation says: "4-byte extension signature. If the first byte is
'A'..'Z' the
extension is optional and can be ignored.". I am getting jibberish. Is
this expected?

Then, the extension size comes back as 35560000 (after ntohl()). That
seems extremely high. My structure is as follows:

struct _indexfile_hdr {
        unsigned char   sig[4];         /* Always "DIRC" */
        uint32_t        version;        /* Version Number */
        uint32_t        entries;        /* Number of extensions */
        unsigned char   extsig[4];      /* Extension signature */
        uint32_t        extsize;        /* Size of the extension */
        uint8_t         sha[8];         /* SHA1 of index before checksum */
} __packed;

Am I doing something wrong? Is there some offset or padding that I missed?

Thanks,
--
Farhan Khan
PGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE

Re: Understanding Index Header

[Kevin Daudt]

On Sat, Oct 06, 2018 at 05:13:45PM -0400, Farhan Khan wrote:
> Hi all,
> 
> I am writing a program that parses out the .git/index file. I am
> reading the git index header documentation, but I seem to be getting
> jibberish data.
> https://github.com/git/git/blob/master/Documentation/technical/index-format.txt
> 
> The first 12 bytes are the signature, version and entries. Great so far.
> 
> Afterwards, I try to read the extension signature, which the
> documentation says: "4-byte extension signature. If the first byte is
> 'A'..'Z' the
> extension is optional and can be ignored.". I am getting jibberish. Is
> this expected?
> 
> Then, the extension size comes back as 35560000 (after ntohl()). That
> seems extremely high. My structure is as follows:
> 
> struct _indexfile_hdr {
>         unsigned char   sig[4];         /* Always "DIRC" */
>         uint32_t        version;        /* Version Number */
>         uint32_t        entries;        /* Number of extensions */
>         unsigned char   extsig[4];      /* Extension signature */
>         uint32_t        extsize;        /* Size of the extension */
>         uint8_t         sha[8];         /* SHA1 of index before checksum */
> } __packed;
> 
> Am I doing something wrong? Is there some offset or padding that I missed?
> 
> Thanks,
> --
> Farhan Khan

Hello Farhan

Between the index header and the extensions there are index entries. So
you have 4 sections:

1. Header (fixed size)
2. Index entries (dynamic size, amount stored in the index)
3. Extensions (dynamic size)
4. hash

So you cannot capture the index with just a single struct. Also what you
call the 'entries' field is not the number of extensions, but the number
of index entries.

I'm not sure why you are seeing so many index entries. I've just tested
it myself on the git repository and I got 3419 entries when reading in
network (big-endian) order.

Hope this helps a bit.

Kevin