* Re: [PATCH] send-email: add --compose-cover option
From: Junio C Hamano @ 2023-10-12 20:31 UTC (permalink / raw)
To: Kristoffer Haugsbakk; +Cc: Ben Dooks, git
In-Reply-To: <277a80c5-40c9-4f1e-a68f-96673380012b@app.fastmail.com>
"Kristoffer Haugsbakk" <code@khaugsbakk.name> writes:
> 2. This would be really useful in `format-patch`. One could write the
> cover letter immediately instead of either (1) editing the generated
> one (placeholders) or (2) providing a file.
> - Personally I don't use `send-email` immediately—I use `format-patch`
> to generate everything so that I can review it before any
> sending. But there are many different workflows.
Hmph, I am not sure why a format-patch user would bother with the
"--compose-cover" option for the cover letter, to be honest. I am a
format-patch user and never drive it from send-email, just like the
workflow you use, and the norm to me is to review everything, not
just the cover letter but also the patches, in my editor session.
You apparently do not need "--edit-patches" option to review and
adjust as needed before sending, even though that is what you do
already. Why do you need a "--compose-cover" option?
^ permalink raw reply
* This list is being migrated to new infrastructure (no action required)
From: Konstantin Ryabitsev @ 2023-10-12 20:30 UTC (permalink / raw)
To: git
Hello, all:
This list is being migrated to the new vger infrastructure. This should be a
fully transparent process and you don't need to change anything about how you
participate with the list or how you receive mail.
There will be a brief delay with archives on lore.kernel.org. I will follow up
once the archive migration has been completed.
Best regards,
Konstantin
^ permalink raw reply
* Re: why does git set X in LESS env var?
From: Christoph Anton Mitterer @ 2023-10-12 20:23 UTC (permalink / raw)
To: Dragan Simic; +Cc: git
In-Reply-To: <161b9584c6c9a004c01bda98cea4f1f8@manjaro.org>
On Thu, 2023-10-12 at 07:46 +0200, Dragan Simic wrote:
> Let me repeat that the messed up output you're experiencing isn't
> normal
> and has nothing to do with the arguments passed to less(1). That's a
> separate issue of the terminal emulator(s) you're using, or in issue
> of
> your specific environment, and should be debugged and addressed as a
> separate issue.
Be it as it may...
As I've told you before it happens at least in gnome-terminal (and thus
presumably and VTE based terminal), xterm, xfce4-terminal and konsole
(all current versions of Debian unstable)... with less as of Debian
unstable as well as 643.
That affects at least on major distro, and there's a good chance that
it affects any other distro based on Debian (*buntu, etc.).
I further tried on SLES 15 with both gnome-terminal 3.42.2 and xterm
330 as well as less 530.
Even tried with the terminal emulator started via env -i and only TERM
set manually.
*All* cases affected by the same problem I've described before.
Same with the command you've used in your follow-up post, here a video
of it in HD:
https://youtu.be/MsxtQgrKM50
> To me, having inconsistent displaying of the short and long outputs
> is
> simply not acceptable.
Which is fine - and as I've said: I personally also tend to prefer it
like that - but even if the above would be just some bug (which however
seems to affect all systems I could test on a short notice, except
yours)... one can IMO still not generally say whether on or the other
behaviour is generally accepted to be the better one.
Even if output may be just chopped of and thus ambiguously incomplete,
some people may still prefer to have rather no output at all.
And in fact:
This is the default mode of less alone.
> >
> Perhaps something is wrong with your specific environment, because I
> see
> no other reason for this issue.
Well may be, but seems unlikely from my PoV, given that I've now tested
even on other distros and systems not under my control.
Anyway... I think this got a bit too off-topic here :-D
Cheers,
Chris.
^ permalink raw reply
* [PATCH] Add x64 SHA-NI implementation to sha1 and sha256 in git ?
From: Feiyang Xue @ 2023-10-12 20:04 UTC (permalink / raw)
To: git; +Cc: Feiyang Xue
Looking for comments on if this is a good idea to add SHA-NI option to git,
And if so, what'd be a good implementation.
The attached patch is more of a proof of concept, using a sha-ni example
found on internet and have it tapped into git's "hash-ll.h" when it sees make
flags "SHA1_SHANI=1" and/or "SHA256_SHANI=1" for sha1/sha256 respectively.
"git hash-object" is about 3-ish times faster for me
There are few topics that i'm uncertain about.
1. Is it good idea to have machine-specific asm codes in git. I read there was
commit fd1ec82547 which removed assembly implementation for PPC_SHA1. Does that
imply maintainers doesn't want machine-specific assembly in source?
2. If it is cool to have machine-specific codes in git, what's a good logic for
their on/off? Build time flag? Run time flag? Run time automatic detection to
corresponding instructions?
3. If i can include machine-specific codes, what'd be a good way/idea to cleanup
the asm? I suppose git maintainers do not want someone else's license header?
Regards,
Feiyang
---
Makefile | 28 ++
hash-ll.h | 8 +-
.../intel_sha_extensions_sha1_assembly.asm | 308 +++++++++++++++
sha1-x64shani/sha1_x64.c | 71 ++++
sha1-x64shani/sha1_x64.h | 21 +
.../intel_sha_extensions_sha256_assembly.asm | 371 ++++++++++++++++++
sha256-x64shani/sha256_x64.c | 82 ++++
sha256-x64shani/sha256_x64.h | 23 ++
8 files changed, 910 insertions(+), 2 deletions(-)
create mode 100644 sha1-x64shani/intel_sha_extensions_sha1_assembly.asm
create mode 100644 sha1-x64shani/sha1_x64.c
create mode 100644 sha1-x64shani/sha1_x64.h
create mode 100644 sha256-x64shani/intel_sha_extensions_sha256_assembly.asm
create mode 100644 sha256-x64shani/sha256_x64.c
create mode 100644 sha256-x64shani/sha256_x64.h
diff --git a/Makefile b/Makefile
index e440728c24..6892af2ee5 100644
--- a/Makefile
+++ b/Makefile
@@ -1917,6 +1917,11 @@ ifdef PPC_SHA1
$(error the PPC_SHA1 flag has been removed along with the PowerPC-specific SHA-1 implementation.)
endif
+ifdef SHA1_SHANI
+ LIB_OBJS += sha1-x64shani/sha1_x64.o
+ LIB_OBJS += sha1-x64shani/intel_sha_extensions_sha1_assembly.o
+ BASIC_CFLAGS += -DSHA1_SHANI
+else
ifdef OPENSSL_SHA1
EXTLIBS += $(LIB_4_CRYPTO)
BASIC_CFLAGS += -DSHA1_OPENSSL
@@ -1957,7 +1962,13 @@ endif
endif
endif
endif
+endif
+ifdef SHA256_SHANI
+ LIB_OBJS += sha256-x64shani/sha256_x64.o
+ LIB_OBJS += sha256-x64shani/intel_sha_extensions_sha256_assembly.o
+ BASIC_CFLAGS += -DSHA256_SHANI
+else
ifdef OPENSSL_SHA256
EXTLIBS += $(LIB_4_CRYPTO)
BASIC_CFLAGS += -DSHA256_OPENSSL
@@ -1975,6 +1986,9 @@ else
endif
endif
endif
+endif
+
+$(info $$LIB_OBJS is [${LIB_OBJS}])
ifdef SHA1_MAX_BLOCK_SIZE
LIB_OBJS += compat/sha1-chunked.o
@@ -2711,6 +2725,20 @@ missing_compdb_dir =
compdb_args =
endif
+$(info $$OBJECTS is [${OBJECTS}])
+
+### for SHA-NI
+YASM_SRC := $(wildcard $(OBJECTS:o=asm))
+YASM_OBJ := $(YASM_SRC:asm=o)
+OBJECTS := $(filter-out $(YASM_OBJ),$(OBJECTS))
+
+$(info $$YASM_SRC is [${YASM_SRC}])
+$(info $$YASM_OBJ is [${YASM_OBJ}])
+$(info $$OBJECTS is [${OBJECTS}])
+
+$(YASM_OBJ): %.o: %.asm $(missing_dep_dirs)
+ $(QUIET_CC)yasm $< -f elf64 -X gnu -g dwarf2 -o $@
+
$(OBJECTS): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)
$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<
diff --git a/hash-ll.h b/hash-ll.h
index 8050925137..31d9e91f26 100644
--- a/hash-ll.h
+++ b/hash-ll.h
@@ -1,7 +1,9 @@
#ifndef HASH_LL_H
#define HASH_LL_H
-#if defined(SHA1_APPLE)
+#if defined(SHA1_SHANI)
+#include "sha1-x64shani/sha1_x64.h"
+#elif defined(SHA1_APPLE)
#include <CommonCrypto/CommonDigest.h>
#elif defined(SHA1_OPENSSL)
#include <openssl/sha.h>
@@ -11,7 +13,9 @@
#include "block-sha1/sha1.h"
#endif
-#if defined(SHA256_NETTLE)
+#if defined(SHA256_SHANI)
+#include "sha256-x64shani/sha256_x64.h"
+#elif defined(SHA256_NETTLE)
#include "sha256/nettle.h"
#elif defined(SHA256_GCRYPT)
#define SHA256_NEEDS_CLONE_HELPER
diff --git a/sha1-x64shani/intel_sha_extensions_sha1_assembly.asm b/sha1-x64shani/intel_sha_extensions_sha1_assembly.asm
new file mode 100644
index 0000000000..ba59825966
--- /dev/null
+++ b/sha1-x64shani/intel_sha_extensions_sha1_assembly.asm
@@ -0,0 +1,308 @@
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+; Copyright (c) 2013, Intel Corporation
+;
+; All rights reserved.
+;
+; Redistribution and use in source and binary forms, with or without
+; modification, are permitted provided that the following conditions are
+; met:
+;
+; * Redistributions of source code must retain the above copyright
+; notice, this list of conditions and the following disclaimer.
+;
+; * Redistributions in binary form must reproduce the above copyright
+; notice, this list of conditions and the following disclaimer in the
+; documentation and/or other materials provided with the
+; distribution.
+;
+; * Neither the name of the Intel Corporation nor the names of its
+; contributors may be used to endorse or promote products derived from
+; this software without specific prior written permission.
+;
+;
+; THIS SOFTWARE IS PROVIDED BY INTEL CORPORATION ""AS IS"" AND ANY
+; EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+; IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+; PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL INTEL CORPORATION OR
+; CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+; EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+; PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+; PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+; LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+; NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+; SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;
+; Intel SHA Extensions optimized implementation of a SHA-1 update function
+;
+; The function takes a pointer to the current hash values, a pointer to the
+; input data, and a number of 64 byte blocks to process. Once all blocks have
+; been processed, the digest pointer is updated with the resulting hash value.
+; The function only processes complete blocks, there is no functionality to
+; store partial blocks. All message padding and hash value initialization must
+; be done outside the update function.
+;
+; The indented lines in the loop are instructions related to rounds processing.
+; The non-indented lines are instructions related to the message schedule.
+;
+; Author: Sean Gulley <sean.m.gulley@intel.com>
+; Date: July 2013
+;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;
+; Example YASM command lines:
+; Windows:
+; yasm -Xvc -f x64 -rnasm -pnasm -DWINABI -o intel_sha_extensions_sha1_assembly.obj -g cv8 intel_sha_extensions_sha1_assembly.asm
+; Linux:
+; yasm -f elf64 -X gnu -g dwarf2 -o intel_sha_extensions_sha1_assembly.o intel_sha_extensions_sha1_assembly.asm
+;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+%ifdef WINABI
+%define DIGEST_PTR rcx ; 1st arg
+%define DATA_PTR rdx ; 2nd arg
+%define NUM_BLKS r8 ; 3rd arg
+%else
+%define DIGEST_PTR rdi ; 1st arg
+%define DATA_PTR rsi ; 2nd arg
+%define NUM_BLKS rdx ; 3rd arg
+%endif
+
+%define RSPSAVE rax
+
+struc frame
+%ifdef WINABI
+.xmm_save: resdq 2
+%endif
+.hash_save: resdq 2
+endstruc
+
+%define ABCD xmm0
+%define E0 xmm1 ; Need two E's b/c they ping pong
+%define E1 xmm2
+%define MSG0 xmm3
+%define MSG1 xmm4
+%define MSG2 xmm5
+%define MSG3 xmm6
+%define SHUF_MASK xmm7
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; void sha1_update(uint32_t *digest, const void *data, uint32_t numBlocks);
+;; arg 1 : pointer to digest
+;; arg 2 : pointer to input data
+;; arg 3 : Number of blocks to process
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+section .text
+global sha1_update:function
+align 32
+sha1_update:
+ mov RSPSAVE, rsp
+ sub rsp, frame_size
+ and rsp, ~0xF
+
+%ifdef WINABI
+ movdqa [rsp + frame.xmm_save + 0*16], xmm6
+ movdqa [rsp + frame.xmm_save + 1*16], xmm7
+%endif
+
+ shl NUM_BLKS, 6 ; convert to bytes
+ jz .done_hash
+ add NUM_BLKS, DATA_PTR ; pointer to end of data
+
+ ;; load initial hash values
+ pinsrd E0, [DIGEST_PTR + 1*16], 3
+ movdqu ABCD, [DIGEST_PTR + 0*16]
+ pand E0, [UPPER_WORD_MASK wrt rip]
+ pshufd ABCD, ABCD, 0x1B
+
+ movdqa SHUF_MASK, [PSHUFFLE_BYTE_FLIP_MASK wrt rip]
+
+.loop0:
+ ;; Save hash values for addition after rounds
+ movdqa [rsp + frame.hash_save + 0*16], E0
+ movdqa [rsp + frame.hash_save + 1*16], ABCD
+
+ ;; Rounds 0-3
+ movdqu MSG0, [DATA_PTR + 0*16]
+ pshufb MSG0, SHUF_MASK
+ paddd E0, MSG0
+ movdqa E1, ABCD
+ sha1rnds4 ABCD, E0, 0
+
+ ;; Rounds 4-7
+ movdqu MSG1, [DATA_PTR + 1*16]
+ pshufb MSG1, SHUF_MASK
+ sha1nexte E1, MSG1
+ movdqa E0, ABCD
+ sha1rnds4 ABCD, E1, 0
+ sha1msg1 MSG0, MSG1
+
+ ;; Rounds 8-11
+ movdqu MSG2, [DATA_PTR + 2*16]
+ pshufb MSG2, SHUF_MASK
+ sha1nexte E0, MSG2
+ movdqa E1, ABCD
+ sha1rnds4 ABCD, E0, 0
+ sha1msg1 MSG1, MSG2
+ pxor MSG0, MSG2
+
+ ;; Rounds 12-15
+ movdqu MSG3, [DATA_PTR + 3*16]
+ pshufb MSG3, SHUF_MASK
+ sha1nexte E1, MSG3
+ movdqa E0, ABCD
+ sha1msg2 MSG0, MSG3
+ sha1rnds4 ABCD, E1, 0
+ sha1msg1 MSG2, MSG3
+ pxor MSG1, MSG3
+
+ ;; Rounds 16-19
+ sha1nexte E0, MSG0
+ movdqa E1, ABCD
+ sha1msg2 MSG1, MSG0
+ sha1rnds4 ABCD, E0, 0
+ sha1msg1 MSG3, MSG0
+ pxor MSG2, MSG0
+
+ ;; Rounds 20-23
+ sha1nexte E1, MSG1
+ movdqa E0, ABCD
+ sha1msg2 MSG2, MSG1
+ sha1rnds4 ABCD, E1, 1
+ sha1msg1 MSG0, MSG1
+ pxor MSG3, MSG1
+
+ ;; Rounds 24-27
+ sha1nexte E0, MSG2
+ movdqa E1, ABCD
+ sha1msg2 MSG3, MSG2
+ sha1rnds4 ABCD, E0, 1
+ sha1msg1 MSG1, MSG2
+ pxor MSG0, MSG2
+
+ ;; Rounds 28-31
+ sha1nexte E1, MSG3
+ movdqa E0, ABCD
+ sha1msg2 MSG0, MSG3
+ sha1rnds4 ABCD, E1, 1
+ sha1msg1 MSG2, MSG3
+ pxor MSG1, MSG3
+
+ ;; Rounds 32-35
+ sha1nexte E0, MSG0
+ movdqa E1, ABCD
+ sha1msg2 MSG1, MSG0
+ sha1rnds4 ABCD, E0, 1
+ sha1msg1 MSG3, MSG0
+ pxor MSG2, MSG0
+
+ ;; Rounds 36-39
+ sha1nexte E1, MSG1
+ movdqa E0, ABCD
+ sha1msg2 MSG2, MSG1
+ sha1rnds4 ABCD, E1, 1
+ sha1msg1 MSG0, MSG1
+ pxor MSG3, MSG1
+
+ ;; Rounds 40-43
+ sha1nexte E0, MSG2
+ movdqa E1, ABCD
+ sha1msg2 MSG3, MSG2
+ sha1rnds4 ABCD, E0, 2
+ sha1msg1 MSG1, MSG2
+ pxor MSG0, MSG2
+
+ ;; Rounds 44-47
+ sha1nexte E1, MSG3
+ movdqa E0, ABCD
+ sha1msg2 MSG0, MSG3
+ sha1rnds4 ABCD, E1, 2
+ sha1msg1 MSG2, MSG3
+ pxor MSG1, MSG3
+
+ ;; Rounds 48-51
+ sha1nexte E0, MSG0
+ movdqa E1, ABCD
+ sha1msg2 MSG1, MSG0
+ sha1rnds4 ABCD, E0, 2
+ sha1msg1 MSG3, MSG0
+ pxor MSG2, MSG0
+
+ ;; Rounds 52-55
+ sha1nexte E1, MSG1
+ movdqa E0, ABCD
+ sha1msg2 MSG2, MSG1
+ sha1rnds4 ABCD, E1, 2
+ sha1msg1 MSG0, MSG1
+ pxor MSG3, MSG1
+
+ ;; Rounds 56-59
+ sha1nexte E0, MSG2
+ movdqa E1, ABCD
+ sha1msg2 MSG3, MSG2
+ sha1rnds4 ABCD, E0, 2
+ sha1msg1 MSG1, MSG2
+ pxor MSG0, MSG2
+
+ ;; Rounds 60-63
+ sha1nexte E1, MSG3
+ movdqa E0, ABCD
+ sha1msg2 MSG0, MSG3
+ sha1rnds4 ABCD, E1, 3
+ sha1msg1 MSG2, MSG3
+ pxor MSG1, MSG3
+
+ ;; Rounds 64-67
+ sha1nexte E0, MSG0
+ movdqa E1, ABCD
+ sha1msg2 MSG1, MSG0
+ sha1rnds4 ABCD, E0, 3
+ sha1msg1 MSG3, MSG0
+ pxor MSG2, MSG0
+
+ ;; Rounds 68-71
+ sha1nexte E1, MSG1
+ movdqa E0, ABCD
+ sha1msg2 MSG2, MSG1
+ sha1rnds4 ABCD, E1, 3
+ pxor MSG3, MSG1
+
+ ;; Rounds 72-75
+ sha1nexte E0, MSG2
+ movdqa E1, ABCD
+ sha1msg2 MSG3, MSG2
+ sha1rnds4 ABCD, E0, 3
+
+ ;; Rounds 76-79
+ sha1nexte E1, MSG3
+ movdqa E0, ABCD
+ sha1rnds4 ABCD, E1, 3
+
+ ;; Add current hash values with previously saved
+ sha1nexte E0, [rsp + frame.hash_save + 0*16]
+ paddd ABCD, [rsp + frame.hash_save + 1*16]
+
+ ;; Increment data pointer and loop if more to process
+ add DATA_PTR, 64
+ cmp DATA_PTR, NUM_BLKS
+ jne .loop0
+
+ ;; Write hash values back in the correct order
+ pshufd ABCD, ABCD, 0x1B
+ movdqu [DIGEST_PTR + 0*16], ABCD
+ pextrd [DIGEST_PTR + 1*16], E0, 3
+
+.done_hash:
+%ifdef WINABI
+ movdqa xmm6, [rsp + frame.xmm_save + 0*16]
+ movdqa xmm7, [rsp + frame.xmm_save + 1*16]
+%endif
+ mov rsp, RSPSAVE
+
+ ret
+
+section .data
+align 64
+PSHUFFLE_BYTE_FLIP_MASK: ddq 0x000102030405060708090a0b0c0d0e0f
+UPPER_WORD_MASK: ddq 0xFFFFFFFF000000000000000000000000
+
diff --git a/sha1-x64shani/sha1_x64.c b/sha1-x64shani/sha1_x64.c
new file mode 100644
index 0000000000..0ed0d49ed6
--- /dev/null
+++ b/sha1-x64shani/sha1_x64.c
@@ -0,0 +1,71 @@
+#include "../git-compat-util.h"
+#include "sha1_x64.h"
+#include <byteswap.h>
+
+// static inline void shani_SHA1_Init(shani_SHA_CTX *c);
+
+void shani_SHA1_Init(shani_SHA_CTX *c)
+{
+ static const uint32_t sha1InitialDigest[5] = {
+ 0x67452301, 0xefcdab89, 0x98badcfe, 0x10325476, 0xc3d2e1f0
+ };
+ memcpy(c->h, sha1InitialDigest, 20);
+ c->len = 0;
+ c->total_len = 0;
+}
+
+void sha1_update(void *c, const void *d, int l);
+void shani_SHA1_Update(shani_SHA_CTX *c, const void *data, unsigned long len)
+{
+ while (len > 0)
+ {
+ if (c->len == 0 && len >= 64)
+ {
+ sha1_update(c->h, data, len/64);
+ unsigned long bytes_hashed = (len/64)*64;
+ data += bytes_hashed;
+ len -= bytes_hashed;
+ c->total_len += bytes_hashed;
+ continue;
+ }
+#define MIN(x, y) (((x) < (y)) ? (x) : (y))
+ size_t cpy = MIN(64 - c->len, len);
+#undef MIN
+ memcpy(c->msgbuf + c->len, data, cpy);
+ data += cpy;
+ len -= cpy;
+ c->len += cpy;
+ c->total_len += cpy;
+
+ if (c->len == 64)
+ {
+ sha1_update(c->h, c->msgbuf, 1);
+ c->len = 0;
+ }
+ }
+}
+
+void shani_SHA1_Final(unsigned char *md, shani_SHA_CTX *c)
+{
+ c->msgbuf[c->len] = 0x80;
+ ++c->len;
+ memset(c->msgbuf + c->len, 0, 64 - c->len);
+ if (c->len > (64-8))
+ {
+ sha1_update(c->h, c->msgbuf, 1);
+ memset(c->msgbuf, 0, 64);
+ c->len = 0;
+ }
+
+ for (int ibyte = 0; ibyte < 8; ++ibyte)
+ c->msgbuf[63-ibyte] = (uint8_t)((c->total_len * 8ULL) >> (ibyte*8));
+
+ sha1_update(c->h, c->msgbuf, 1);
+ c->len = 0;
+
+ // flip endianness
+ int i;
+ for (i = 0; i < 20; i += 4) {
+ *(uint32_t *)(md + i) = __bswap_32(c->h[i / 4]);
+ }
+}
diff --git a/sha1-x64shani/sha1_x64.h b/sha1-x64shani/sha1_x64.h
new file mode 100644
index 0000000000..ca1ec116cf
--- /dev/null
+++ b/sha1-x64shani/sha1_x64.h
@@ -0,0 +1,21 @@
+/*
+ * SHA1 routine using x64 SHA-NI instructions
+ */
+
+typedef struct shani_SHA_CTX
+{
+ uint32_t h[5];
+ int len;
+ uint64_t total_len;
+ char msgbuf[64];
+} shani_SHA_CTX;
+
+
+void shani_SHA1_Init(shani_SHA_CTX *ctx);
+void shani_SHA1_Update(shani_SHA_CTX *ctx, const void *dataIn, size_t len);
+void shani_SHA1_Final(unsigned char hashout[20], shani_SHA_CTX *ctx);
+
+#define platform_SHA_CTX shani_SHA_CTX
+#define platform_SHA1_Init shani_SHA1_Init
+#define platform_SHA1_Update shani_SHA1_Update
+#define platform_SHA1_Final shani_SHA1_Final
diff --git a/sha256-x64shani/intel_sha_extensions_sha256_assembly.asm b/sha256-x64shani/intel_sha_extensions_sha256_assembly.asm
new file mode 100644
index 0000000000..b587e82154
--- /dev/null
+++ b/sha256-x64shani/intel_sha_extensions_sha256_assembly.asm
@@ -0,0 +1,371 @@
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+; Copyright (c) 2013, Intel Corporation
+;
+; All rights reserved.
+;
+; Redistribution and use in source and binary forms, with or without
+; modification, are permitted provided that the following conditions are
+; met:
+;
+; * Redistributions of source code must retain the above copyright
+; notice, this list of conditions and the following disclaimer.
+;
+; * Redistributions in binary form must reproduce the above copyright
+; notice, this list of conditions and the following disclaimer in the
+; documentation and/or other materials provided with the
+; distribution.
+;
+; * Neither the name of the Intel Corporation nor the names of its
+; contributors may be used to endorse or promote products derived from
+; this software without specific prior written permission.
+;
+;
+; THIS SOFTWARE IS PROVIDED BY INTEL CORPORATION ""AS IS"" AND ANY
+; EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+; IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+; PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL INTEL CORPORATION OR
+; CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+; EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+; PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+; PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+; LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+; NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+; SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;
+; Intel SHA Extensions optimized implementation of a SHA-256 update function
+;
+; The function takes a pointer to the current hash values, a pointer to the
+; input data, and a number of 64 byte blocks to process. Once all blocks have
+; been processed, the digest pointer is updated with the resulting hash value.
+; The function only processes complete blocks, there is no functionality to
+; store partial blocks. All message padding and hash value initialization must
+; be done outside the update function.
+;
+; The indented lines in the loop are instructions related to rounds processing.
+; The non-indented lines are instructions related to the message schedule.
+;
+; Author: Sean Gulley <sean.m.gulley@intel.com>
+; Date: July 2013
+;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;
+; Example YASM command lines:
+; Windows: yasm -Xvc -f x64 -rnasm -pnasm -DWINABI -o intel_sha_extensions_sha256_assembly.obj -g cv8 intel_sha_extensions_sha256_assembly.asm
+; Linux: yasm -f elf64 -X gnu -g dwarf2 -o intel_sha_extensions_sha256_assembly.o intel_sha_extensions_sha256_assembly.asm
+;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+%ifdef WINABI
+%define DIGEST_PTR rcx ; 1st arg
+%define DATA_PTR rdx ; 2nd arg
+%define NUM_BLKS r8 ; 3rd arg
+%else
+%define DIGEST_PTR rdi ; 1st arg
+%define DATA_PTR rsi ; 2nd arg
+%define NUM_BLKS rdx ; 3rd arg
+%endif
+
+%define SHA256CONSTANTS rax
+
+%ifdef WINABI
+%define RSPSAVE r9
+
+struc frame
+.xmm_save: resdq 5
+endstruc
+%endif
+
+%define MSG xmm0
+%define STATE0 xmm1
+%define STATE1 xmm2
+%define MSGTMP0 xmm3
+%define MSGTMP1 xmm4
+%define MSGTMP2 xmm5
+%define MSGTMP3 xmm6
+%define MSGTMP4 xmm7
+
+%define SHUF_MASK xmm8
+
+%define ABEF_SAVE xmm9
+%define CDGH_SAVE xmm10
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; void sha256_update(uint32_t *digest, const void *data, uint32_t numBlocks);
+;; arg 1 : pointer to digest
+;; arg 2 : pointer to input data
+;; arg 3 : Number of blocks to process
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+section .text
+global sha256_update:function
+align 32
+sha256_update:
+%ifdef WINABI
+ mov RSPSAVE, rsp
+ sub rsp, frame_size
+ and rsp, ~0xF
+
+ movdqa [rsp + frame.xmm_save + 0*16], xmm6
+ movdqa [rsp + frame.xmm_save + 1*16], xmm7
+ movdqa [rsp + frame.xmm_save + 2*16], xmm8
+ movdqa [rsp + frame.xmm_save + 3*16], xmm9
+ movdqa [rsp + frame.xmm_save + 4*16], xmm10
+%endif
+
+ shl NUM_BLKS, 6 ; convert to bytes
+ jz .done_hash
+ add NUM_BLKS, DATA_PTR ; pointer to end of data
+
+ ;; load initial hash values
+ ;; Need to reorder these appropriately
+ ;; DCBA, HGFE -> ABEF, CDGH
+ movdqu STATE0, [DIGEST_PTR + 0*16]
+ movdqu STATE1, [DIGEST_PTR + 1*16]
+
+ pshufd STATE0, STATE0, 0xB1 ; CDAB
+ pshufd STATE1, STATE1, 0x1B ; EFGH
+ movdqa MSGTMP4, STATE0
+ palignr STATE0, STATE1, 8 ; ABEF
+ pblendw STATE1, MSGTMP4, 0xF0 ; CDGH
+
+ movdqa SHUF_MASK, [PSHUFFLE_BYTE_FLIP_MASK wrt rip]
+ lea SHA256CONSTANTS,[K256 wrt rip]
+
+.loop0:
+ ;; Save hash values for addition after rounds
+ movdqa ABEF_SAVE, STATE0
+ movdqa CDGH_SAVE, STATE1
+
+ ;; Rounds 0-3
+ movdqu MSG, [DATA_PTR + 0*16]
+ pshufb MSG, SHUF_MASK
+ movdqa MSGTMP0, MSG
+ paddd MSG, [SHA256CONSTANTS + 0*16]
+ sha256rnds2 STATE1, STATE0
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+
+ ;; Rounds 4-7
+ movdqu MSG, [DATA_PTR + 1*16]
+ pshufb MSG, SHUF_MASK
+ movdqa MSGTMP1, MSG
+ paddd MSG, [SHA256CONSTANTS + 1*16]
+ sha256rnds2 STATE1, STATE0
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP0, MSGTMP1
+
+ ;; Rounds 8-11
+ movdqu MSG, [DATA_PTR + 2*16]
+ pshufb MSG, SHUF_MASK
+ movdqa MSGTMP2, MSG
+ paddd MSG, [SHA256CONSTANTS + 2*16]
+ sha256rnds2 STATE1, STATE0
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP1, MSGTMP2
+
+ ;; Rounds 12-15
+ movdqu MSG, [DATA_PTR + 3*16]
+ pshufb MSG, SHUF_MASK
+ movdqa MSGTMP3, MSG
+ paddd MSG, [SHA256CONSTANTS + 3*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP3
+ palignr MSGTMP4, MSGTMP2, 4
+ paddd MSGTMP0, MSGTMP4
+ sha256msg2 MSGTMP0, MSGTMP3
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP2, MSGTMP3
+
+ ;; Rounds 16-19
+ movdqa MSG, MSGTMP0
+ paddd MSG, [SHA256CONSTANTS + 4*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP0
+ palignr MSGTMP4, MSGTMP3, 4
+ paddd MSGTMP1, MSGTMP4
+ sha256msg2 MSGTMP1, MSGTMP0
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP3, MSGTMP0
+
+ ;; Rounds 20-23
+ movdqa MSG, MSGTMP1
+ paddd MSG, [SHA256CONSTANTS + 5*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP1
+ palignr MSGTMP4, MSGTMP0, 4
+ paddd MSGTMP2, MSGTMP4
+ sha256msg2 MSGTMP2, MSGTMP1
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP0, MSGTMP1
+
+ ;; Rounds 24-27
+ movdqa MSG, MSGTMP2
+ paddd MSG, [SHA256CONSTANTS + 6*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP2
+ palignr MSGTMP4, MSGTMP1, 4
+ paddd MSGTMP3, MSGTMP4
+ sha256msg2 MSGTMP3, MSGTMP2
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP1, MSGTMP2
+
+ ;; Rounds 28-31
+ movdqa MSG, MSGTMP3
+ paddd MSG, [SHA256CONSTANTS + 7*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP3
+ palignr MSGTMP4, MSGTMP2, 4
+ paddd MSGTMP0, MSGTMP4
+ sha256msg2 MSGTMP0, MSGTMP3
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP2, MSGTMP3
+
+ ;; Rounds 32-35
+ movdqa MSG, MSGTMP0
+ paddd MSG, [SHA256CONSTANTS + 8*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP0
+ palignr MSGTMP4, MSGTMP3, 4
+ paddd MSGTMP1, MSGTMP4
+ sha256msg2 MSGTMP1, MSGTMP0
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP3, MSGTMP0
+
+ ;; Rounds 36-39
+ movdqa MSG, MSGTMP1
+ paddd MSG, [SHA256CONSTANTS + 9*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP1
+ palignr MSGTMP4, MSGTMP0, 4
+ paddd MSGTMP2, MSGTMP4
+ sha256msg2 MSGTMP2, MSGTMP1
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP0, MSGTMP1
+
+ ;; Rounds 40-43
+ movdqa MSG, MSGTMP2
+ paddd MSG, [SHA256CONSTANTS + 10*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP2
+ palignr MSGTMP4, MSGTMP1, 4
+ paddd MSGTMP3, MSGTMP4
+ sha256msg2 MSGTMP3, MSGTMP2
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP1, MSGTMP2
+
+ ;; Rounds 44-47
+ movdqa MSG, MSGTMP3
+ paddd MSG, [SHA256CONSTANTS + 11*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP3
+ palignr MSGTMP4, MSGTMP2, 4
+ paddd MSGTMP0, MSGTMP4
+ sha256msg2 MSGTMP0, MSGTMP3
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP2, MSGTMP3
+
+ ;; Rounds 48-51
+ movdqa MSG, MSGTMP0
+ paddd MSG, [SHA256CONSTANTS + 12*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP0
+ palignr MSGTMP4, MSGTMP3, 4
+ paddd MSGTMP1, MSGTMP4
+ sha256msg2 MSGTMP1, MSGTMP0
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+ sha256msg1 MSGTMP3, MSGTMP0
+
+ ;; Rounds 52-55
+ movdqa MSG, MSGTMP1
+ paddd MSG, [SHA256CONSTANTS + 13*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP1
+ palignr MSGTMP4, MSGTMP0, 4
+ paddd MSGTMP2, MSGTMP4
+ sha256msg2 MSGTMP2, MSGTMP1
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+
+ ;; Rounds 56-59
+ movdqa MSG, MSGTMP2
+ paddd MSG, [SHA256CONSTANTS + 14*16]
+ sha256rnds2 STATE1, STATE0
+ movdqa MSGTMP4, MSGTMP2
+ palignr MSGTMP4, MSGTMP1, 4
+ paddd MSGTMP3, MSGTMP4
+ sha256msg2 MSGTMP3, MSGTMP2
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+
+ ;; Rounds 60-63
+ movdqa MSG, MSGTMP3
+ paddd MSG, [SHA256CONSTANTS + 15*16]
+ sha256rnds2 STATE1, STATE0
+ pshufd MSG, MSG, 0x0E
+ sha256rnds2 STATE0, STATE1
+
+ ;; Add current hash values with previously saved
+ paddd STATE0, ABEF_SAVE
+ paddd STATE1, CDGH_SAVE
+
+ ;; Increment data pointer and loop if more to process
+ add DATA_PTR, 64
+ cmp DATA_PTR, NUM_BLKS
+ jne .loop0
+
+ ;; Write hash values back in the correct order
+ pshufd STATE0, STATE0, 0x1B ; FEBA
+ pshufd STATE1, STATE1, 0xB1 ; DCHG
+ movdqa MSGTMP4, STATE0
+ pblendw STATE0, STATE1, 0xF0 ; DCBA
+ palignr STATE1, MSGTMP4, 8 ; HGFE
+
+ movdqu [DIGEST_PTR + 0*16], STATE0
+ movdqu [DIGEST_PTR + 1*16], STATE1
+
+.done_hash:
+%ifdef WINABI
+ movdqa xmm6, [rsp + frame.xmm_save + 0*16]
+ movdqa xmm7, [rsp + frame.xmm_save + 1*16]
+ movdqa xmm8, [rsp + frame.xmm_save + 2*16]
+ movdqa xmm9, [rsp + frame.xmm_save + 3*16]
+ movdqa xmm10, [rsp + frame.xmm_save + 4*16]
+ mov rsp, RSPSAVE
+%endif
+
+ ret
+
+section .data
+align 64
+K256:
+ dd 0x428a2f98,0x71374491,0xb5c0fbcf,0xe9b5dba5
+ dd 0x3956c25b,0x59f111f1,0x923f82a4,0xab1c5ed5
+ dd 0xd807aa98,0x12835b01,0x243185be,0x550c7dc3
+ dd 0x72be5d74,0x80deb1fe,0x9bdc06a7,0xc19bf174
+ dd 0xe49b69c1,0xefbe4786,0x0fc19dc6,0x240ca1cc
+ dd 0x2de92c6f,0x4a7484aa,0x5cb0a9dc,0x76f988da
+ dd 0x983e5152,0xa831c66d,0xb00327c8,0xbf597fc7
+ dd 0xc6e00bf3,0xd5a79147,0x06ca6351,0x14292967
+ dd 0x27b70a85,0x2e1b2138,0x4d2c6dfc,0x53380d13
+ dd 0x650a7354,0x766a0abb,0x81c2c92e,0x92722c85
+ dd 0xa2bfe8a1,0xa81a664b,0xc24b8b70,0xc76c51a3
+ dd 0xd192e819,0xd6990624,0xf40e3585,0x106aa070
+ dd 0x19a4c116,0x1e376c08,0x2748774c,0x34b0bcb5
+ dd 0x391c0cb3,0x4ed8aa4a,0x5b9cca4f,0x682e6ff3
+ dd 0x748f82ee,0x78a5636f,0x84c87814,0x8cc70208
+ dd 0x90befffa,0xa4506ceb,0xbef9a3f7,0xc67178f2
+
+PSHUFFLE_BYTE_FLIP_MASK: ddq 0x0c0d0e0f08090a0b0405060700010203
+
diff --git a/sha256-x64shani/sha256_x64.c b/sha256-x64shani/sha256_x64.c
new file mode 100644
index 0000000000..01f3417719
--- /dev/null
+++ b/sha256-x64shani/sha256_x64.c
@@ -0,0 +1,82 @@
+#include "../git-compat-util.h"
+#include "sha256_x64.h"
+#include <byteswap.h>
+
+static int shani = -1;
+
+void shani_SHA256_Init(shani_SHA256_CTX *ctx)
+{
+ ctx->total_len = 0;
+ ctx->len = 0;
+ ctx->h[0] = 0x6a09e667ul;
+ ctx->h[1] = 0xbb67ae85ul;
+ ctx->h[2] = 0x3c6ef372ul;
+ ctx->h[3] = 0xa54ff53aul;
+ ctx->h[4] = 0x510e527ful;
+ ctx->h[5] = 0x9b05688cul;
+ ctx->h[6] = 0x1f83d9abul;
+ ctx->h[7] = 0x5be0cd19ul;
+
+ if (shani != 1) {
+ shani = 1;
+ printf("set shani to 1\n");
+ } else {
+ printf("shani is 1 already\n");
+ }
+}
+
+void sha256_update(uint32_t *digest, const void *data, uint32_t numBlocks);
+void shani_SHA256_Update(shani_SHA256_CTX *c, const void *data, size_t len)
+{
+ while (len > 0)
+ {
+ if (c->len == 0 && len >= 64)
+ {
+ sha256_update(c->h, data, len/64);
+ unsigned long bytes_hashed = (len/64)*64;
+ data += bytes_hashed;
+ len -= bytes_hashed;
+ c->total_len += bytes_hashed;
+ continue;
+ }
+#define MIN(x, y) (((x) < (y)) ? (x) : (y))
+ size_t cpy = MIN(64 - c->len, len);
+#undef MIN
+ memcpy(c->msgbuf + c->len, data, cpy);
+ data += cpy;
+ len -= cpy;
+ c->len += cpy;
+ c->total_len += cpy;
+
+ if (c->len == 64)
+ {
+ sha256_update(c->h, c->msgbuf, 1);
+ c->len = 0;
+ }
+ }
+}
+
+void shani_SHA256_Final(unsigned char *digest, shani_SHA256_CTX *c)
+{
+ c->msgbuf[c->len] = 0x80;
+ ++c->len;
+ memset(c->msgbuf + c->len, 0, 64 - c->len);
+ if (c->len > (64-8))
+ {
+ sha256_update(c->h, c->msgbuf, 1);
+ memset(c->msgbuf, 0, 64);
+ c->len = 0;
+ }
+
+ for (int ibyte = 0; ibyte < 8; ++ibyte)
+ c->msgbuf[63-ibyte] = (uint8_t)((c->total_len * 8ULL) >> (ibyte*8));
+
+ sha256_update(c->h, c->msgbuf, 1);
+ c->len = 0;
+
+ // flip endianness
+ int i;
+ for (i = 0; i < 32; i += 4) {
+ *(uint32_t *)(digest + i) = __bswap_32(c->h[i / 4]);
+ }
+}
\ No newline at end of file
diff --git a/sha256-x64shani/sha256_x64.h b/sha256-x64shani/sha256_x64.h
new file mode 100644
index 0000000000..25f1ff67ba
--- /dev/null
+++ b/sha256-x64shani/sha256_x64.h
@@ -0,0 +1,23 @@
+/*
+ * SHA256 routine using x64 SHA-NI instructions
+ */
+
+#define SHANI_SHA256_BLKSIZE 64
+
+struct shani_SHA256_CTX {
+ uint32_t h[8];
+ int len;
+ uint64_t total_len;
+ char msgbuf[SHANI_SHA256_BLKSIZE];
+};
+
+typedef struct shani_SHA256_CTX shani_SHA256_CTX;
+
+void shani_SHA256_Init(shani_SHA256_CTX *ctx);
+void shani_SHA256_Update(shani_SHA256_CTX *c, const void *data, size_t len);
+void shani_SHA256_Final(unsigned char *digest, shani_SHA256_CTX *c);
+
+#define platform_SHA256_CTX shani_SHA256_CTX
+#define platform_SHA256_Init shani_SHA256_Init
+#define platform_SHA256_Update shani_SHA256_Update
+#define platform_SHA256_Final shani_SHA256_Final
\ No newline at end of file
--
2.41.0
^ permalink raw reply related
* Re: [PATCH] send-email: add --compose-cover option
From: Kristoffer Haugsbakk @ 2023-10-12 20:03 UTC (permalink / raw)
To: Ben Dooks; +Cc: git, Junio C Hamano
In-Reply-To: <20231012112743.2756259-1-ben.dooks@codethink.co.uk>
Hi Ben
On Thu, Oct 12, 2023, at 13:27, Ben Dooks wrote:
> Adding an option to automatically compose a cover letter would be
> helpful to put the whole process of sending an series with a cover
> into one command.
I didn't manage to apply the patch so I haven't tried it out. Some
questions:
1. There is already `--compose` which is clearly different. What's the
difference between a cover letter an an introduction? That's a more
immediate question now with two similar-looking options.
2. This would be really useful in `format-patch`. One could write the
cover letter immediately instead of either (1) editing the generated
one (placeholders) or (2) providing a file.
- Personally I don't use `send-email` immediately—I use `format-patch`
to generate everything so that I can review it before any
sending. But there are many different workflows.
3. Does this editor only deal with the subject and the message part before
the diffstat and all the auto-generated stuff? And then it applies it
after the user closes the editor (to make the complete cover letter)?
--
Kristoffer Haugsbakk
^ permalink raw reply
* -q option for git to suppress informational messages?
From: Michal Suchánek @ 2023-10-12 18:53 UTC (permalink / raw)
To: git
Hello,
when using git in scripts I find that many git commands insist on
printing informational messages, and the only way to avoid this is to
sent the output to /dev/null.
While some select commands support the -q option to suppres
informational messages many don't.
Since there was recenly some discussion of problems that newcomers could
work on I suppose this could be added to the list if not there already.
Thanks
Michal
^ permalink raw reply
* git and multiple processes
From: Federico Kircheis @ 2023-10-12 18:28 UTC (permalink / raw)
To: git
Hello,
I wanted to know if concurrent command can "break" a repository, ie put
it in an inconsistent state.
I suppose that as long all commands are only querying the status (log,
reflog, status, ....) then such a thing will not happen.
I guess that if only one command changes the status (add, commit, pull,
fetch, push, ...) then the output might be "wrong", but the command that
queries the status does not interfere with the others.
But what if two commands that change the status?
For example a git fetch and git add or git commit at the same time?
A use-case I had recently was me doing "git commit -p" from one console,
and noticing a change I wanted to undo.
I skipped the change, and while "git commit -p" showed me the next diff,
I opened another console.
From there, I did a "git checkout", and reverted the change I just skipped.
Is such use-case supported?
Everything worked as I expected, but it would be nice to know that I did
not "risk" anything.
The use-cases I am most interested in is when working with worktrees.
What if I add a worktree, and "work concurrently" in two worktrees?
For example while git is switching branches in one worktree, doing a
commit in another worktree (assuming I'm not switching to the branch
where I am committing)
What if I am doing an automated "git bisect" in a worktree.
Can I work in another one (doing commits, fetch, rebase, reflog, stash,
...)?
What if I have a cron job that executes "git fetch" in one worktree.
Is it problematic if I happen to execute git fetch at the same time from
another worktree?
If those use-cases are not problematic, as all repositories are linked
together, what are possible issues (ignoring bugs).
Common sense tells me that those operations are supported, but I could
not find any reference, or if there are some guarantees that git wants
to hold, no matter what the current status is.
Best
Federico
^ permalink raw reply
* Re: [PATCH v2 2/2] Prevent git from rehashing 4GiB files
From: Junio C Hamano @ 2023-10-12 17:58 UTC (permalink / raw)
To: brian m. carlson; +Cc: git, Taylor Blau, Jason Hatton
In-Reply-To: <20231012160930.330618-3-sandals@crustytoothpaste.net>
"brian m. carlson" <sandals@crustytoothpaste.net> writes:
> An example would be to have a 2^32 sized file in the index of
> patched git. Patched git would save the file as 2^31 in the cache.
> An unpatched git would very much see the file has changed in size
> and force it to rehash the file, which is safe.
The reason why this is "safe" is because an older Git will would
keep rehashing whether 2^31 or 0 is stored as its sd_size, so the
change is not making things worse? With older git, "git diff-files"
will report that such a file is not up to date, and then the user
will refresh the index, which will store 0 as its sd_file, so
tentatively "git status" may give a wrong information, but we
probalby do not care? Is that how the reasoning goes?
> +/*
> + * Munge st_size into an unsigned int.
> + */
> +static unsigned int munge_st_size(off_t st_size) {
> + unsigned int sd_size = st_size;
> +
> + /*
> + * If the file is an exact multiple of 4 GiB, modify the value so it
> + * doesn't get marked as racily clean (zero).
> + */
> + if (!sd_size && st_size)
> + return 0x80000000;
> + else
> + return sd_size;
> +}
This assumes typeof(sd_size) aka "unsigned int" is always 32-bit,
which does not sound reasonable. Reference to 4GiB, 2^32 and 2^31
in the code and the proposed commit log message need to be qualified
with "on a platform whose uint is 32-bit" or something, or better
yet, phrased in a way that is agnostic to the integer size. At
the very least, the hardcoded 0x80000000 needs to be rethought, I
am afraid.
Other than that, the workaround for the racy-git avoidance code does
sound good. I actually wonder if we should always use 1 regardless
of integer size.
^ permalink raw reply
* Re: [PATCH v2 1/2] t: add a test helper to truncate files
From: Eric Sunshine @ 2023-10-12 17:49 UTC (permalink / raw)
To: brian m. carlson; +Cc: git, Junio C Hamano, Taylor Blau, Jason Hatton
In-Reply-To: <20231012160930.330618-2-sandals@crustytoothpaste.net>
On Thu, Oct 12, 2023 at 12:10 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> In a future commit, we're going to work with some large files which will
> be at least 4 GiB in size. To take advantage of the sparseness
> functionality on most Unix systems and avoid running the system out of
> disk, it would be convenient to use truncate(2) to simply create a
> sparse file of sufficient size.
>
> However, the GNU truncate(1) utility isn't portable, so let's write a
> tiny test helper that does the work for us.
>
> Signed-off-by: brian m. carlson <bk2204@github.com>
> ---
> diff --git a/t/helper/test-truncate.c b/t/helper/test-truncate.c
> @@ -0,0 +1,27 @@
> +int cmd__truncate(int argc, const char **argv)
> +{
> + char *p = NULL;
> + uintmax_t sz = 0;
> + int fd = -1;
> +
> + if (argc != 3)
> + die("expected filename and size");
> +
> + sz = strtoumax(argv[2], &p, 0);
> + if (*p)
> + die("invalid size");
Do you want to check 'errno' here, as well (probably before the '*p' check)?
Or is that being too defensive for a 'test-tool' command?
> + fd = open(argv[1], O_WRONLY | O_CREAT, 0600);
> + if (fd < 0)
> + die_errno("failed to open file %s", argv[1]);
> +
> + if (ftruncate(fd, (off_t) sz) < 0)
> + die_errno("failed to truncate file");
> + return 0;
> +}
^ permalink raw reply
* Re: [PATCH] mailmap: change primary address for Derrick Stolee
From: Junio C Hamano @ 2023-10-12 17:46 UTC (permalink / raw)
To: Derrick Stolee via GitGitGadget; +Cc: git, Derrick Stolee
In-Reply-To: <pull.1592.git.1697131834003.gitgitgadget@gmail.com>
[jc: removed @github.com address from CC]
"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> -Derrick Stolee <derrickstolee@github.com> <stolee@gmail.com>
> -Derrick Stolee <derrickstolee@github.com> Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
> -Derrick Stolee <derrickstolee@github.com> <dstolee@microsoft.com>
> +Derrick Stolee <stolee@gmail.com> <derrickstolee@github.com>
> +Derrick Stolee <stolee@gmail.com> Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
> +Derrick Stolee <stolee@gmail.com> <dstolee@microsoft.com>
It has been irritating to see @github.com address bouncing. Will
apply.
Thanks.
^ permalink raw reply
* Re: Bug: Git grep -f reads the filename relative to the repository root
From: Junio C Hamano @ 2023-10-12 17:28 UTC (permalink / raw)
To: Erik Cervin Edin; +Cc: Git Mailing List
In-Reply-To: <CA+JQ7M_htKUv5=GRQEUqWmJrQmFQNfZkPjr8n12CU6x0Khr4dw@mail.gmail.com>
Erik Cervin Edin <erik@cervined.in> writes:
> In the Git repository, I ran
>
> echo tig > pattern-file &&
> echo git > xdiff/pattern-file &&
> cd xdfiff &&
> git grep -f pattern-file
>
> What did you expect to happen? (Expected behavior)
>
> Git grep -f to read the pattern-file, in the xdiff directory and
> search for lines matching `git` in the xdiff directory.
That does sound like a bug. It should use the original directory as
the base of the relative path computation, similar to the way how
OPT_FILENAME() options are handled.
Perhaps something along this line, but this is not even compile
tested yet.
----- >8 --------- >8 --------- >8 --------- >8 -----
Subject: [PATCH] grep: -f <path> is relative to $cwd
Just like OPT_FILENAME() does, "git grep -f <path>" should treat
the <path> relative to the original $cwd by paying attention to the
prefix the command is given.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
builtin/grep.c | 13 +++++++++++--
t/t7810-grep.sh | 13 +++++++++++++
2 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/builtin/grep.c b/builtin/grep.c
index b71222330a..fe78d4c98b 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -4,6 +4,7 @@
* Copyright (c) 2006 Junio C Hamano
*/
#include "builtin.h"
+#include "abspath.h"
#include "gettext.h"
#include "hex.h"
#include "repository.h"
@@ -812,14 +813,20 @@ static int file_callback(const struct option *opt, const char *arg, int unset)
{
struct grep_opt *grep_opt = opt->value;
int from_stdin;
+ const char *filename = arg;
FILE *patterns;
int lno = 0;
struct strbuf sb = STRBUF_INIT;
BUG_ON_OPT_NEG(unset);
- from_stdin = !strcmp(arg, "-");
- patterns = from_stdin ? stdin : fopen(arg, "r");
+ if (!*filename)
+ ; /* leave it as-is */
+ else
+ filename = prefix_filename_except_for_dash(grep_prefix, filename);
+
+ from_stdin = !strcmp(filename, "-");
+ patterns = from_stdin ? stdin : fopen(filename, "r");
if (!patterns)
die_errno(_("cannot open '%s'"), arg);
while (strbuf_getline(&sb, patterns) == 0) {
@@ -833,6 +840,8 @@ static int file_callback(const struct option *opt, const char *arg, int unset)
if (!from_stdin)
fclose(patterns);
strbuf_release(&sb);
+ if (filename != arg)
+ free((void *)filename);
return 0;
}
diff --git a/t/t7810-grep.sh b/t/t7810-grep.sh
index 39d6d713ec..91ac66935f 100755
--- a/t/t7810-grep.sh
+++ b/t/t7810-grep.sh
@@ -808,6 +808,19 @@ test_expect_success 'grep -f, ignore empty lines, read patterns from stdin' '
test_cmp expected actual
'
+test_expect_success 'grep -f, use cwd relative file' '
+ test_when_finished "git rm -f sub/dir/file" &&
+ mkdir -p sub/dir &&
+ echo hit >sub/dir/file &&
+ git add sub/dir/file &&
+ echo hit >sub/dir/pattern &&
+ echo miss >pattern &&
+ (
+ cd sub/dir && git grep -f pattern file
+ ) &&
+ git -C sub/dir grep -f pattern file
+'
+
cat >expected <<EOF
y:y yy
--
--
2.42.0-345-gaab89be2eb
^ permalink raw reply related
* [PATCH] mailmap: change primary address for Derrick Stolee
From: Derrick Stolee via GitGitGadget @ 2023-10-12 17:30 UTC (permalink / raw)
To: git; +Cc: gitster, Derrick Stolee, Derrick Stolee
From: Derrick Stolee <stolee@gmail.com>
The previous primary address is no longer valid.
Signed-off-by: Derrick Stolee <stolee@gmail.com>
---
mailmap: change primary address for Derrick Stolee
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1592%2Fderrickstolee%2Fmailmap-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1592/derrickstolee/mailmap-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1592
.mailmap | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/.mailmap b/.mailmap
index dc31d70b8c1..82129be449f 100644
--- a/.mailmap
+++ b/.mailmap
@@ -59,9 +59,9 @@ David Reiss <dreiss@facebook.com> <dreiss@dreiss-vmware.(none)>
David S. Miller <davem@davemloft.net>
David Turner <novalis@novalis.org> <dturner@twopensource.com>
David Turner <novalis@novalis.org> <dturner@twosigma.com>
-Derrick Stolee <derrickstolee@github.com> <stolee@gmail.com>
-Derrick Stolee <derrickstolee@github.com> Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
-Derrick Stolee <derrickstolee@github.com> <dstolee@microsoft.com>
+Derrick Stolee <stolee@gmail.com> <derrickstolee@github.com>
+Derrick Stolee <stolee@gmail.com> Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
+Derrick Stolee <stolee@gmail.com> <dstolee@microsoft.com>
Deskin Miller <deskinm@umich.edu>
Đoàn Trần Công Danh <congdanhqx@gmail.com> Doan Tran Cong Danh
Dirk Süsserott <newsletter@dirk.my1.cc>
base-commit: bcb6cae2966cc407ca1afc77413b3ef11103c175
--
gitgitgadget
^ permalink raw reply related
* Re: How to combine multiple commit diffs?
From: Johannes Sixt @ 2023-10-12 16:41 UTC (permalink / raw)
To: ZheNing Hu; +Cc: Git List, Junio C Hamano
In-Reply-To: <CAOLTT8RzcENBx9NKffHReVKJAho89TCO7W2SPBX8sb2tEU84Gw@mail.gmail.com>
[For general support questions, please address only the mailing list.
It's not necessary to bother the maintainer personally. Though, I'll not
remove him from Cc, yet, as to comply with this ML's etiquette.]
Am 12.10.23 um 14:00 schrieb ZheNing Hu:
> Hi everyone,
>
> Our company wants to design a "small-batch" code review feature.
> Simply put, this "small-batch" means being able to treat multiple
> related commits within a MergeRequest as an independent "small" code
> review.
>
> Let me give you an example: We have five commits: A1, B, A2, C, A3.
> Among them, A1, A2, and A3 are multiple commits for the same feature.
> So when the user selects these commits, the page will return a
> "combine diff" that combines them together.
>
> A1 B A2 A3 C
> *--------*----*-----*-------* (branch)
> \ A1' \ A2' \ A3'
> *------------*------*------- (small branch code review)
>
> This may seem similar to cherry-picking a few commits from a pile of
> commits, but in fact, we do not expect to actually perform
> cherry-picking.
>
> Do you have any suggestions on how we can merge a few commits together
> and display the diff? The only reference we have is the non-open
> source platform, JetBrains Space CodeReview, they support selecting
> multiple commits for CodeReview. [1], .
Take a step back. Then ask: What are the consequences of the review?
What if the result is: the feature is perfect, we want it merged,
however, we cannot, because we do not want commit B. What if the result
is the opposite? You need B, but you can't merge it because the feature
is not ready, yet?
You are looking for a technical workaround for a non-optimal workflow.
If A1,A2,A3 are a feature on their own, they, and only they, should be
in their own feature branch.
So, I would say, the best solution is to reorder the commits in a better
manageable order. You do know about git rebase --interactive, don't you?
-- Hannes
^ permalink raw reply
* Re: [Outreachy]Introduction and Problem while installing Git
From: Dorcas Litunya @ 2023-10-12 17:13 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <xmqqmswnhkxt.fsf@gitster.g>
On Thu, Oct 12, 2023 at 09:20:14AM -0700, Junio C Hamano wrote:
> Benson Muite <benson_muite@emailplus.org> writes:
>
> > On 10/12/23 09:57, Dorcas Litunya wrote:
> >> Hello everyone,
> >> My name is Dorcas Litunya. I am excited to contribute to the git
> >> community, I am a first time contributor through the Outreachy program.
> >> I am excited to learn and grow through this project. I am currently
> >> installing Git and I have been faced with this error once I run the make
> >> command:
> >> In file included from http.c:2:
> >> git-curl-compat.h:3:10: fatal error: curl/curl.h: No such file or directory
> >> 3 | #include <curl/curl.h>
> >> | ^~~~~~~~~~~~~
> > You will need to have curl libraries and development headers.
> > https://curl.se/libcurl/
> > You maybe able to get these from a package manager, for example on Ubuntu
> > sudo apt-get install curl-dev
> > Fedora
> > sudo dnf install libcurl-devel
>
> Thanks for helping. Perhaps reading the INSTALL file at the top of
> the working tree would worth the time?
Thanks Junio. I did follow it eventually. I got stuck here since the
instructions to install some libraries was at the towards the middle and
end of the INSTALL file while the
instruction to run the make command was at the top of the file, and it
seemed a bit easy as the file had just said "it should work
normally"(without pointing out the libraries needed) was what kinda got
me lost, but I eventually found my solutions as I read through the doc. But yes,I will now try to read and search entire docs as all the good stuff might be at the bottom.
:-)
>
^ permalink raw reply
* Re: How to combine multiple commit diffs?
From: Junio C Hamano @ 2023-10-12 17:03 UTC (permalink / raw)
To: ZheNing Hu; +Cc: Git List
In-Reply-To: <CAOLTT8RzcENBx9NKffHReVKJAho89TCO7W2SPBX8sb2tEU84Gw@mail.gmail.com>
ZheNing Hu <adlternative@gmail.com> writes:
> This may seem similar to cherry-picking a few commits from a pile of
> commits, but in fact, we do not expect to actually perform
> cherry-picking.
If you said "We do not want to", I may be sympathetic, but it is
curious that you said "We do not expect to", which hints that you
already have some implementation in mind.
Whether you use a checkout of the history to a working tree and
perform a series of "git cherry-pick" to implement it, or use the
"git replay" machinery to perform them in-core, at the conceptual
level, you would need to pick a base commit and apply the effect of
those commits you would want to look at on top of that base in
sequence, before you can see the combined change, no?
Puzzled.
^ permalink raw reply
* Re: [RFC] Define "precious" attribute and support it in `git clean`
From: Junio C Hamano @ 2023-10-12 16:58 UTC (permalink / raw)
To: Sebastian Thiel; +Cc: git, Josh Triplett, Kristoffer Haugsbakk
In-Reply-To: <0E44CB2C-57F2-4075-95BE-60FBFDD3CEE2@icloud.com>
Sebastian Thiel <sebastian.thiel@icloud.com> writes:
> ### What about a `$` syntax in `.gitignore` files?
>
> I looked into adding a new prefix, `$` to indicate the following path is
> precious or… valuable. It can be escaped with `\$` just like `\!`.
I have been regretting that I did not make the quoting syntax not
obviously extensible in f87f9497 (git-ls-files: --exclude mechanism
updates., 2005-07-24), which technically was a breaking change (as a
relative pathname that began with '!' were not special, but after
the change, it became necessary to '\'-quote it). A relative
pathname that begins with '$' would be now broken the same way, but
hopefully the fallout would be minor. I presume you picked '$'
exactly because of this reason?
I do not think it will be the end of the world if we don't do so,
but it would be really really nice if we at least explored a way (or
two) to make a big enough hole in the syntax to not just add
"precious", but leave room to later add other traits, without having
to worry about breaking the backward compatibility again. A
simplest and suboptimal way may be to declare that a path that
begins with '$' now needs '\'-quoting (just like your proposal),
reserve '$$' as the precious prefix, and '$' followed by any other
byte reserved for future use, but there may be better ideas.
> *Unfortunately*, users can't just add a local `.git/info/exclude` file with
> `$.config` in it and expect `.config` to be considered precious as the pattern
> search order will search this last as it's part of the exclude-globals.
That it nothing new and is the same for ignored files. The lower
precedence files do not override higher precedence files.
> Thus, to make this work, projects that ship the `.gitignore` files would *have
> to add patterns* that make certain files precious.
Not really. They do not have to do anything if they are content
with the current Git ecosystem. And users who have precious stuff
can mark them in the.git/info/excludes no? The only case that is
problematic is when the project says 'foo' is ignored and expendable
but the user thinks otherwise. So to make this work, projects that
ship the ".gitignore" files have to avoid adding patterns to ignore
things that it may reasonably be expected for its users to mark
precious.
> Such opted-in projects would produce `.gitignore` files like these:
>
> .*
> $.config
I would understand if you ignored "*~" or "*.o", but why ignore ".*"?
THanks.
^ permalink raw reply
* Re: [PATCH 0/3] rev-list: add support for commits in `--missing`
From: Junio C Hamano @ 2023-10-12 16:26 UTC (permalink / raw)
To: Karthik Nayak; +Cc: Patrick Steinhardt, git
In-Reply-To: <CAOLa=ZQxNX4oGtqrgLyKenC_D8M=9q0sFJVmo4fyjSPtgw315Q@mail.gmail.com>
Karthik Nayak <karthik.188@gmail.com> writes:
> ... To ensure that the feature should
> work with corrupt repositories with stale commit-graph, I'm thinking of
> disabling the commit-graph whenever the `--missing` option is used.
Just to avoid giving anybody a wrong guideline, for most of the
other features, we should assume that the commit-graph accurately
represents the reality in the object database and take advantage of
it, leaving it to "git fsck" to ensure that the commit-graph is
healthy.
But "rev-list --missing", especially when used with actions other
than allow-promisor or allow-any, is a feature that is only useful
when one wants to really notice objects that are available in the
object store, and it would be a bug for it to pretend as if objects
the commit-graph has heard of exists without checking.
^ permalink raw reply
* Re: [Outreachy]Introduction and Problem while installing Git
From: Junio C Hamano @ 2023-10-12 16:20 UTC (permalink / raw)
To: Benson Muite; +Cc: Dorcas Litunya, git
In-Reply-To: <4c5fef38-a671-dd6b-4b10-a531e1ae254a@emailplus.org>
Benson Muite <benson_muite@emailplus.org> writes:
> On 10/12/23 09:57, Dorcas Litunya wrote:
>> Hello everyone,
>> My name is Dorcas Litunya. I am excited to contribute to the git
>> community, I am a first time contributor through the Outreachy program.
>> I am excited to learn and grow through this project. I am currently
>> installing Git and I have been faced with this error once I run the make
>> command:
>> In file included from http.c:2:
>> git-curl-compat.h:3:10: fatal error: curl/curl.h: No such file or directory
>> 3 | #include <curl/curl.h>
>> | ^~~~~~~~~~~~~
> You will need to have curl libraries and development headers.
> https://curl.se/libcurl/
> You maybe able to get these from a package manager, for example on Ubuntu
> sudo apt-get install curl-dev
> Fedora
> sudo dnf install libcurl-devel
Thanks for helping. Perhaps reading the INSTALL file at the top of
the working tree would worth the time?
^ permalink raw reply
* Re: why does git set X in LESS env var?
From: Junio C Hamano @ 2023-10-12 16:19 UTC (permalink / raw)
To: Dragan Simic; +Cc: Jeff King, Christoph Anton Mitterer, git
In-Reply-To: <3946c06e90604a92ad0dddf787729668@manjaro.org>
Dragan Simic <dsimic@manjaro.org> writes:
> Please note that dropping "-X" and leaving "-F" would actually
> introduce the inconsistency that I already mentioned. To reiterate,
> short outputs would then remain displayed on screen, while long
> outputs would disappear after exiting less(1).
Good point.
^ permalink raw reply
* Re: [PATCH 0/3] rev-list: add support for commits in `--missing`
From: Junio C Hamano @ 2023-10-12 16:17 UTC (permalink / raw)
To: Patrick Steinhardt; +Cc: Karthik Nayak, git
In-Reply-To: <ZSfSt4tXx8sE68Bn@tanuki>
Patrick Steinhardt <ps@pks.im> writes:
> Wouldn't this have the potential to significantly regress performance
> for all those preexisting users of the `--missing` option? The commit
> graph is quite an important optimization nowadays, and especially in
> commands where we potentially walk a lot of commits (like we may do
> here) it can result in queries that are orders of magnitudes faster.
The test fails only when GIT_TEST_COMMIT_GRAPH is on, which updates
the commit-graph every time a commit is made via "git commit" or
"git merge".
I'd suggest stepping back and think a bit.
My assumption has been that the failing test emulates this scenario
that can happen in real life:
* The user creates a new commit.
* A commit graph is written (not as part of GIT_TEST_COMMIT_GRAPH
that is not realistic, but as part of "maintenance").
* The repository loses some objects due to corruption.
* Now, "--missing=print" is invoked so that the user can view what
are missing. Or "--missing=allow-primisor" to ensure that the
repository does not have missing objects other than the ones that
the promisor will give us if we asked again.
* But because the connectivity of these objects appear in the
commit graph file, we fail to notice that these objects are
missing, producing wrong results. If we disabled commit-graph
while traversal (an earlier writing of it was perfectly OK), then
"rev-list --missing" would have noticed and reported what the
user wanted to know.
In other words, the "optimization" you value is working to quickly
produce a wrong result. Is it "significantly regress"ing if we
disabled it to obtain the correct result?
My assumption also has been that there is no point in running
"rev-list --missing" if we know there is no repository corruption,
and those who run "rev-list --missing" wants to know if the objects
are really available, i.e. even if commit-graph that is out of sync
with reality says it exists, if it is not in the object store, they
would want to know that.
If you can show me that it is not the case, then I may be pursuaded
why producing a result that is out of sync with reality _quickly_,
instead of taking time to produce a result that matches reality, is
a worthy "optimization" to keep.
Thanks.
^ permalink raw reply
* [PATCH v2 0/2] Prevent re-reading 4 GiB files on every status
From: brian m. carlson @ 2023-10-12 16:09 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Taylor Blau, Jason Hatton
Several people have noticed that Git continually re-reads and re-hashes
files that are an exact multiple of 4 GiB every time the index is
refreshed (for example, when "git status" is called). This is slow and
expensive, especially with SHA-1 DC, and it also causes performance
problems when these files are used with Git LFS (because the same issue
occurs, just with Git LFS hashing the data instead of Git).
Jason Hatton sent a patch previously to fix this, but it lacked tests
and didn't get picked up. I've adopted their patch, making some minor
changes to the commit message and including some tests, and also
including a suitable test helper to make the tests possible. All credit
should be directed to Jason, and I'll accept all the responsibility for
any problems.
I don't anticipate this being in any way controversial, so I'm not
expecting a huge number of rerolls, but of course one or two might be
necessary.
Jason Hatton (1):
Prevent git from rehashing 4GiB files
brian m. carlson (1):
t: add a test helper to truncate files
Makefile | 1 +
statinfo.c | 20 ++++++++++++++++++--
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
t/helper/test-truncate.c | 27 +++++++++++++++++++++++++++
t/t7508-status.sh | 16 ++++++++++++++++
6 files changed, 64 insertions(+), 2 deletions(-)
create mode 100644 t/helper/test-truncate.c
^ permalink raw reply
* [PATCH v2 2/2] Prevent git from rehashing 4GiB files
From: brian m. carlson @ 2023-10-12 16:09 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Taylor Blau, Jason Hatton
In-Reply-To: <20231012160930.330618-1-sandals@crustytoothpaste.net>
From: Jason Hatton <jhatton@globalfinishing.com>
The index stores file sizes using a uint32_t. This causes any file
that is a multiple of 2^32 to have a cached file size of zero.
Zero is a special value used by racily clean. This causes git to
rehash every file that is a multiple of 2^32 every time git status
or git commit is run.
This patch mitigates the problem by making all files that are a
multiple of 2^32 appear to have a size of 1<<31 instead of zero.
The value of 1<<31 is chosen to keep it as far away from zero
as possible to help prevent things getting mixed up with unpatched
versions of git.
An example would be to have a 2^32 sized file in the index of
patched git. Patched git would save the file as 2^31 in the cache.
An unpatched git would very much see the file has changed in size
and force it to rehash the file, which is safe. The file would
have to grow or shrink by exactly 2^31 and retain all of its
ctime, mtime, and other attributes for old git to not notice
the change.
This patch does not change the behavior of any file that is not
an exact multiple of 2^32.
Signed-off-by: Jason D. Hatton <jhatton@globalfinishing.com>
Signed-off-by: brian m. carlson <bk2204@github.com>
---
statinfo.c | 20 ++++++++++++++++++--
t/t7508-status.sh | 16 ++++++++++++++++
2 files changed, 34 insertions(+), 2 deletions(-)
diff --git a/statinfo.c b/statinfo.c
index 17bb8966c3..9367ca099c 100644
--- a/statinfo.c
+++ b/statinfo.c
@@ -2,6 +2,22 @@
#include "environment.h"
#include "statinfo.h"
+/*
+ * Munge st_size into an unsigned int.
+ */
+static unsigned int munge_st_size(off_t st_size) {
+ unsigned int sd_size = st_size;
+
+ /*
+ * If the file is an exact multiple of 4 GiB, modify the value so it
+ * doesn't get marked as racily clean (zero).
+ */
+ if (!sd_size && st_size)
+ return 0x80000000;
+ else
+ return sd_size;
+}
+
void fill_stat_data(struct stat_data *sd, struct stat *st)
{
sd->sd_ctime.sec = (unsigned int)st->st_ctime;
@@ -12,7 +28,7 @@ void fill_stat_data(struct stat_data *sd, struct stat *st)
sd->sd_ino = st->st_ino;
sd->sd_uid = st->st_uid;
sd->sd_gid = st->st_gid;
- sd->sd_size = st->st_size;
+ sd->sd_size = munge_st_size(st->st_size);
}
int match_stat_data(const struct stat_data *sd, struct stat *st)
@@ -51,7 +67,7 @@ int match_stat_data(const struct stat_data *sd, struct stat *st)
changed |= INODE_CHANGED;
#endif
- if (sd->sd_size != (unsigned int) st->st_size)
+ if (sd->sd_size != munge_st_size(st->st_size))
changed |= DATA_CHANGED;
return changed;
diff --git a/t/t7508-status.sh b/t/t7508-status.sh
index 6928fd89f5..6c46648e11 100755
--- a/t/t7508-status.sh
+++ b/t/t7508-status.sh
@@ -1745,4 +1745,20 @@ test_expect_success 'slow status advice when core.untrackedCache true, and fsmon
)
'
+test_expect_success EXPENSIVE 'status does not re-read unchanged 4 or 8 GiB file' '
+ (
+ mkdir large-file &&
+ cd large-file &&
+ # Files are 2 GiB, 4 GiB, and 8 GiB sparse files.
+ test-tool truncate file-a 0x080000000 &&
+ test-tool truncate file-b 0x100000000 &&
+ test-tool truncate file-c 0x200000000 &&
+ # This will be slow.
+ git add file-a file-b file-c &&
+ git commit -m "add large files" &&
+ git diff-index HEAD file-a file-b file-c >actual &&
+ test_must_be_empty actual
+ )
+'
+
test_done
^ permalink raw reply related
* [PATCH v2 1/2] t: add a test helper to truncate files
From: brian m. carlson @ 2023-10-12 16:09 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Taylor Blau, Jason Hatton
In-Reply-To: <20231012160930.330618-1-sandals@crustytoothpaste.net>
From: "brian m. carlson" <bk2204@github.com>
In a future commit, we're going to work with some large files which will
be at least 4 GiB in size. To take advantage of the sparseness
functionality on most Unix systems and avoid running the system out of
disk, it would be convenient to use truncate(2) to simply create a
sparse file of sufficient size.
However, the GNU truncate(1) utility isn't portable, so let's write a
tiny test helper that does the work for us.
Signed-off-by: brian m. carlson <bk2204@github.com>
---
Makefile | 1 +
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
t/helper/test-truncate.c | 27 +++++++++++++++++++++++++++
4 files changed, 30 insertions(+)
create mode 100644 t/helper/test-truncate.c
diff --git a/Makefile b/Makefile
index 9c6a2f125f..03adcb5a48 100644
--- a/Makefile
+++ b/Makefile
@@ -852,6 +852,7 @@ TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o
TEST_BUILTINS_OBJS += test-submodule.o
TEST_BUILTINS_OBJS += test-subprocess.o
TEST_BUILTINS_OBJS += test-trace2.o
+TEST_BUILTINS_OBJS += test-truncate.o
TEST_BUILTINS_OBJS += test-urlmatch-normalization.o
TEST_BUILTINS_OBJS += test-userdiff.o
TEST_BUILTINS_OBJS += test-wildmatch.o
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index 9010ac6de7..876cd2dc31 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -86,6 +86,7 @@ static struct test_cmd cmds[] = {
{ "submodule-nested-repo-config", cmd__submodule_nested_repo_config },
{ "subprocess", cmd__subprocess },
{ "trace2", cmd__trace2 },
+ { "truncate", cmd__truncate },
{ "userdiff", cmd__userdiff },
{ "urlmatch-normalization", cmd__urlmatch_normalization },
{ "xml-encode", cmd__xml_encode },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index f134f96b97..70dd4eba11 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -79,6 +79,7 @@ int cmd__submodule_config(int argc, const char **argv);
int cmd__submodule_nested_repo_config(int argc, const char **argv);
int cmd__subprocess(int argc, const char **argv);
int cmd__trace2(int argc, const char **argv);
+int cmd__truncate(int argc, const char **argv);
int cmd__userdiff(int argc, const char **argv);
int cmd__urlmatch_normalization(int argc, const char **argv);
int cmd__xml_encode(int argc, const char **argv);
diff --git a/t/helper/test-truncate.c b/t/helper/test-truncate.c
new file mode 100644
index 0000000000..bd3fde364b
--- /dev/null
+++ b/t/helper/test-truncate.c
@@ -0,0 +1,27 @@
+#include "test-tool.h"
+#include "git-compat-util.h"
+
+/*
+ * Truncate a file to the given size.
+ */
+int cmd__truncate(int argc, const char **argv)
+{
+ char *p = NULL;
+ uintmax_t sz = 0;
+ int fd = -1;
+
+ if (argc != 3)
+ die("expected filename and size");
+
+ sz = strtoumax(argv[2], &p, 0);
+ if (*p)
+ die("invalid size");
+
+ fd = open(argv[1], O_WRONLY | O_CREAT, 0600);
+ if (fd < 0)
+ die_errno("failed to open file %s", argv[1]);
+
+ if (ftruncate(fd, (off_t) sz) < 0)
+ die_errno("failed to truncate file");
+ return 0;
+}
^ permalink raw reply related
* Re: [PATCH 0/3] rev-list: add support for commits in `--missing`
From: Karthik Nayak @ 2023-10-12 13:23 UTC (permalink / raw)
To: Patrick Steinhardt; +Cc: Junio C Hamano, git
In-Reply-To: <ZSfSt4tXx8sE68Bn@tanuki>
On Thu, Oct 12, 2023 at 1:04 PM Patrick Steinhardt <ps@pks.im> wrote:
> Wouldn't this have the potential to significantly regress performance
> for all those preexisting users of the `--missing` option? The commit
> graph is quite an important optimization nowadays, and especially in
> commands where we potentially walk a lot of commits (like we may do
> here) it can result in queries that are orders of magnitudes faster.
>
> If we were introducing a new option then I think this would be an
> acceptable tradeoff as we could still fix the performance issue in
> another iteration. But we don't and instead aim to change the meaning of
> `--missing` which may already have users out there. Seen in that light
> it does not seem sensible to me to just disable the graph for them.
>
Agreed, that there is a rather huge performance drop doing this. So either:
1. Add commit support for `--missing` and disable commit-graph. There is huge
performance drop here for commits.
2. Add commit support for `--missing` but disabling commit-graph is provided via
an additional `--disable-commit-graph` option. This expects the user to manually
disable commit-graph for `--missing` to work correctly with commits.
I was also thinking about this, but people who are using `--missing`
till date, also use
it with `--objects`. Using `--objects` is already slow and doesn't
benefit from the
commit-graph. So I don't know if there is much of a performance hit.
Some benchmarks on the Git repo:
With commit graphs:
$ hyperfine --warmup 2 "git rev-list --objects --missing=print HEAD > /dev/null"
Benchmark 1: git rev-list --objects --missing=print HEAD > /dev/null
Time (mean ± σ): 1.336 s ± 0.013 s [User: 1.278 s, System: 0.054 s]
Range (min … max): 1.323 s … 1.365 s 10 runs
$ hyperfine --warmup 2 "git rev-list --missing=print HEAD > /dev/null"
Benchmark 1: git rev-list --missing=print HEAD > /dev/null
Time (mean ± σ): 29.6 ms ± 1.5 ms [User: 20.2 ms, System: 9.1 ms]
Range (min … max): 27.5 ms … 35.1 ms 92 runs
Without commit graphs:
$ hyperfine --warmup 2 "git rev-list --objects --missing=print HEAD > /dev/null"
Benchmark 1: git rev-list --objects --missing=print HEAD > /dev/null
Time (mean ± σ): 1.735 s ± 0.022 s [User: 1.672 s, System: 0.057 s]
Range (min … max): 1.703 s … 1.765 s 10 runs
$ hyperfine --warmup 2 "git rev-list --missing=print HEAD > /dev/null"
Benchmark 1: git rev-list --missing=print HEAD > /dev/null
Time (mean ± σ): 426.6 ms ± 10.2 ms [User: 410.1 ms, System: 15.1 ms]
Range (min … max): 414.9 ms … 441.3 ms 10 runs
This shows that while there is definitely a performance loss when
disabling commit-graphs.
This loss is much lesser when used alongside the `--objects` flag.
Overall, I lean towards the option "1", but okay with either.
> Did you figure out what exactly the root cause of this functional
> regression is? If so, can we address that root cause instead?
Are you referring to `--missing` not working with commits? Looking at
the history
of `--missing` shows:
- caf3827e2f: This commits introduces the `--missing` option, it
basically states that
for the upcoming partial clone mechanism, we want to add support for identifying
missing objects.
- 7c0fe330d5: This commit adds support for trees in `--missing`. It
also goes on to
state that `--missing` assumed only blobs could be missing.
Both of these show that `--missing` started off support for only blobs
and eventually
added support to trees (similar to how we're adding support for
commits here). So
I guess the intention was to never work with commits. But the documentation and
the option seem generic enough. Considering that trees was an extension, commits
should be treated the same way.
^ permalink raw reply
* Re: [PATCH] send-email: add --compose-cover option
From: Ben Dooks @ 2023-10-12 13:07 UTC (permalink / raw)
To: git, gitster
In-Reply-To: <20231012112743.2756259-1-ben.dooks@codethink.co.uk>
On 12/10/2023 12:27, Ben Dooks wrote:
> Adding an option to automatically compose a cover letter would be
> helpful to put the whole process of sending an series with a cover
> into one command.
>
> Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
> ---
> Documentation/git-send-email.txt | 5 +++++
> git-send-email.perl | 11 ++++++++++-
> 2 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/git-send-email.txt b/Documentation/git-send-email.txt
> index 41cd8cb424..f299732867 100644
> --- a/Documentation/git-send-email.txt
> +++ b/Documentation/git-send-email.txt
> @@ -78,6 +78,11 @@ Missing From or In-Reply-To headers will be prompted for.
> +
> See the CONFIGURATION section for `sendemail.multiEdit`.
>
> +--compose-cover::
> + Invoke a text editor (see GIT_EDITOR in linkgit:git-var[1])
> + to edit a cover letter generated by passing --cover-letter to
> + git-send-email before invoking the editor.
> +
OOPS, clearly meant git-format-patch here, not git-send-email.
> --from=<address>::
> Specify the sender of the emails. If not specified on the command line,
> the value of the `sendemail.from` configuration option is used. If
> diff --git a/git-send-email.perl b/git-send-email.perl
> index 5861e99a6e..debec088f6 100755
> --- a/git-send-email.perl
> +++ b/git-send-email.perl
> @@ -54,6 +54,7 @@ sub usage {
> --in-reply-to <str> * Email "In-Reply-To:"
> --[no-]xmailer * Add "X-Mailer:" header (default).
> --[no-]annotate * Review each patch that will be sent in an editor.
> + --compose-cover * Open an editor on format-patch --cover-letter
> --compose * Open an editor for introduction.
> --compose-encoding <str> * Encoding to assume for introduction.
> --8bit-encoding <str> * Encoding to assume 8bit mails if undeclared
> @@ -199,7 +200,7 @@ sub format_2822_time {
> # Variables we fill in automatically, or via prompting:
> my (@to,@cc,@xh,$envelope_sender,
> $initial_in_reply_to,$reply_to,$initial_subject,@files,
> - $author,$sender,$smtp_authpass,$annotate,$compose,$time);
> + $author,$sender,$smtp_authpass,$annotate,$compose_cover,$compose,$time);
> # Things we either get from config, *or* are overridden on the
> # command-line.
> my ($no_cc, $no_to, $no_bcc, $no_identity);
> @@ -512,6 +513,7 @@ sub config_regexp {
> "no-smtp-auth" => sub {$smtp_auth = 'none'},
> "annotate!" => \$annotate,
> "no-annotate" => sub {$annotate = 0},
> + "compose-cover" => \$compose_cover,
> "compose" => \$compose,
> "quiet" => \$quiet,
> "cc-cmd=s" => \$cc_cmd,
> @@ -782,6 +784,9 @@ sub is_format_patch_arg {
> die __("Cannot run git format-patch from outside a repository\n")
> unless $repo;
> require File::Temp;
> + if ($compose_cover) {
> + push @rev_list_opts, "--cover-letter";
> + }
> push @files, $repo->command('format-patch', '-o', File::Temp::tempdir(CLEANUP => 1), @rev_list_opts);
> }
>
> @@ -854,6 +859,8 @@ sub get_patch_subject {
>
> if ($annotate) {
> do_edit($compose_filename, @files);
> + } elsif ($compose_cover) {
> + do_edit($files[0]);
> } else {
> do_edit($compose_filename);
> }
> @@ -927,6 +934,8 @@ sub get_patch_subject {
>
> } elsif ($annotate) {
> do_edit(@files);
> +} elsif ($compose_cover) {
> + do_edit($files[0]);
> }
>
> sub term {
--
Ben Dooks http://www.codethink.co.uk/
Senior Engineer Codethink - Providing Genius
https://www.codethink.co.uk/privacy.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox