* What's cooking in git.git (Nov 2014, #04; Wed, 26)
@ 2014-11-26 23:09 Junio C Hamano
2014-11-27 5:24 ` [PATCH] pack-bitmap: do not use gcc packed attribute Jeff King
2014-11-27 18:39 ` What's cooking in git.git (Nov 2014, #04; Wed, 26) brian m. carlson
0 siblings, 2 replies; 13+ messages in thread
From: Junio C Hamano @ 2014-11-26 23:09 UTC (permalink / raw)
To: git
Here are the topics that have been cooking. Commits prefixed with
'-' are only in 'pu' (proposed updates) while commits prefixed with
'+' are in 'next'.
I haven't had time to write comments (they are essentially release
notes entries to be used when/if the topic graduates to 'master')
for new topics, but I am pushing this out to show the current state
of affairs.
You can find the changes described here in the integration branches
of the repositories listed at
http://git-blame.blogspot.com/p/git-public-repositories.html
--------------------------------------------------
[Graduated to "master"]
* cc/interpret-trailers (2014-11-10) 2 commits
(merged to 'next' on 2014-11-14 at fa0ccc6)
+ trailer: display a trailer without its trailing newline
+ trailer: ignore comment lines inside the trailers
(this branch is used by cc/interpret-trailers-more.)
Small fixes to a new experimental command already in 'master'.
* da/difftool (2014-11-14) 1 commit
(merged to 'next' on 2014-11-17 at 231f559)
+ difftool: honor --trust-exit-code for builtin tools
Fix-up to a new feature in 'master'.
* jc/doc-commit-only (2014-11-07) 1 commit
(merged to 'next' on 2014-11-14 at 29c70d2)
+ Documentation/git-commit: clarify that --only/--include records the working tree contents
* mh/doc-remote-helper-xref (2014-11-11) 1 commit
(merged to 'next' on 2014-11-14 at a7f6230)
+ doc: add some crossrefs between manual pages
* sn/tutorial-status-output-example (2014-11-13) 1 commit
(merged to 'next' on 2014-11-14 at 0fe4930)
+ gittutorial: fix output of 'git status'
* sv/submitting-final-patch (2014-11-13) 1 commit
(merged to 'next' on 2014-11-14 at 9e1220f)
+ SubmittingPatches: final submission is To: maintainer and CC: list
* ta/tutorial-modernize (2014-11-11) 1 commit
(merged to 'next' on 2014-11-14 at c45f0ac)
+ gittutorial.txt: remove reference to ancient Git version
* tb/no-relative-file-url (2014-11-13) 1 commit
(merged to 'next' on 2014-11-14 at 96e9227)
+ t5705: the file:// URL should be absolute
--------------------------------------------------
[New Topics]
* jc/unpack-trees-plug-leak (2014-11-17) 1 commit
- unpack_trees: plug leakage of o->result
* js/windows-open-eisdir-error (2014-11-17) 1 commit
(merged to 'next' on 2014-11-18 at 57b0d49)
+ Windows: correct detection of EISDIR in mingw_open()
* rs/maint-config-use-labs (2014-11-17) 1 commit
(merged to 'next' on 2014-11-18 at 53c2404)
+ use labs() for variables of type long instead of abs()
* rs/receive-pack-use-labs (2014-11-17) 1 commit
(merged to 'next' on 2014-11-18 at c6d2d94)
+ use labs() for variables of type long instead of abs()
* jk/colors (2014-11-20) 5 commits
- diff-highlight: allow configurable colors
- parse_color: recognize "no$foo" to clear the $foo attribute
- parse_color: support 24-bit RGB values
- parse_color: refactor color storage
- Merge branch 'jn/parse-config-slot' into jk/colors
(this branch uses jk/colors-fix.)
* jk/colors-fix (2014-11-20) 3 commits
- t4026: test "normal" color
- config: fix parsing of "git config --get-color some.key -1"
- docs: describe ANSI 256-color mode
(this branch is used by jk/colors.)
* jk/gitweb-with-newer-cgi-multi-param (2014-11-18) 1 commit
(merged to 'next' on 2014-11-18 at 6ac61fe)
+ gitweb: hack around CGI's list-context param() handling
* jk/lock-ref-sha1-basic-return-errors (2014-11-20) 1 commit
- lock_ref_sha1_basic: do not die on locking errors
* jk/no-perl-tests (2014-11-18) 2 commits
- t960[34]: mark cvsimport tests as requiring perl
- t0090: mark add-interactive test with PERL prerequisite
* jk/rebuild-perl-scripts-with-no-perl-seting-change (2014-11-18) 3 commits
- Makefile: have python scripts depend on NO_PYTHON setting
- Makefile: simplify by using SCRIPT_{PERL,SH}_GEN macros
- Makefile: have perl scripts depend on NO_PERL setting
* mh/config-copy-string-from-git-path (2014-11-17) 1 commit
- cmd_config(): make a copy of path obtained from git_path()
* mh/config-flip-xbit-back-after-checking (2014-11-18) 1 commit
(merged to 'next' on 2014-11-18 at 45f7d71)
+ create_default_files(): don't set u+x bit on $GIT_DIR/config
(this branch is used by tb/config-core-filemode-check-on-broken-fs.)
* po/everyday-doc (2014-11-17) 1 commit
- Documentation: change "gitlink" typo in git-push
* ps/new-workdir-into-empty-directory (2014-11-20) 1 commit
- git-new-workdir: Don't fail if the target directory is empty
* rt/push-recurse-submodule-usage-string (2014-11-18) 1 commit
- builtin/push.c: fix description of --recurse-submodules option
* sb/copy-fd-errno (2014-11-17) 1 commit
- copy.c: make copy_fd preserve meaningful errno
* sb/log-ref-write-fd (2014-11-20) 1 commit
- refs.c: add a function to append a reflog entry to a fd
* sb/ref-transaction-unify-to-update (2014-11-20) 2 commits
- refs.c: make ref_transaction_delete a wrapper for ref_transaction_update
- refs.c: make ref_transaction_create a wrapper for ref_transaction_update
* sv/doc-stripspace (2014-11-19) 1 commit
- Documentation/git-stripspace: Update synopsis
* sv/typofix-apply-error-message (2014-11-17) 1 commit
- apply: fix typo in an error message
* da/difftool-mergetool-simplify-reporting-status (2014-11-21) 5 commits
- mergetools: stop setting $status in merge_cmd()
- mergetool: simplify conditionals
- difftool--helper: add explicit exit statement
- mergetool--lib: remove use of $status global
- mergetool--lib: remove no-op assignment to $status from setup_user_tool
* dw/shell-basename-dashdash-before-stripping-leading-dash-from-login (2014-11-25) 1 commit
- git-sh-setup.sh: use dashdash with basename call
* jc/refer-to-t-readme-from-submitting-patches (2014-11-24) 2 commits
- t/README: justify why "! grep foo" is sufficient
- SubmittingPatches: refer to t/README for tests
* jc/t9001-modernise (2014-11-25) 5 commits
- t9001: style modernisation phase #5
- t9001: style modernisation phase #4
- t9001: style modernisation phase #3
- t9001: style modernisation phase #2
- t9001: style modernisation phase #1
(this branch uses pb/send-email-te.)
* js/t5000-dont-copy-bin-sh (2014-11-24) 1 commit
- t5000 on Windows: do not mistake "sh.exe" as "sh"
* mg/add-ignore-errors (2014-11-21) 1 commit
- add: ignore only ignored files
* mh/find-uniq-abbrev (2014-11-26) 1 commit
- sha1_name: avoid unnecessary sha1 lookup in find_unique_abbrev
* mh/simplify-repack-without-refs (2014-11-25) 7 commits
- sort_string_list(): rename to string_list_sort()
- prune_remote(): iterate using for_each_string_list_item()
- prune_remote(): rename local variable
- repack_without_refs(): make the refnames argument a string_list
- prune_remote(): sort delete_refs_list references en masse
- prune_remote(): initialize both delete_refs lists in a single loop
- prune_remote(): exit early if there are no stale references
* pb/am-message-id-footer (2014-11-25) 2 commits
- git-am: add --message-id/--no-message-id
- git-mailinfo: add --message-id
* pb/send-email-te (2014-11-25) 2 commits
- git-send-email: add --transfer-encoding option
- git-send-email: delay creation of MIME headers
(this branch is used by jc/t9001-modernise.)
* pw/remote-set-url-fetch (2014-11-26) 1 commit
- remote: add --fetch and --both options to set-url
* rj/no-xopen-source-for-cygwin (2014-11-24) 1 commit
- git-compat-util.h: don't define _XOPEN_SOURCE on cygwin
* sb/string-list (2014-11-25) 3 commits
- string_list: remove string_list_insert_at_index() from its API
- mailmap: use higher level string list functions
- string_list: document string_list_(insert,lookup)
* sb/write-sha1-update-reflog (2014-11-24) 1 commit
- refs.c: move reflog updates into its own function
* tb/config-core-filemode-check-on-broken-fs (2014-11-21) 1 commit
- init-db: improve the filemode trustability check
(this branch uses mh/config-flip-xbit-back-after-checking.)
--------------------------------------------------
[Stalled]
* je/quiltimport-no-fuzz (2014-10-21) 2 commits
- git-quiltimport: flip the default not to allow fuzz
- git-quiltimport.sh: allow declining fuzz with --exact option
"quiltimport" drove "git apply" always with -C1 option to reduce
context of the patch in order to give more chance to somewhat stale
patches to apply. Add an "--exact" option to disable, and also
"-C$n" option to customize this behaviour. The top patch
optionally flips the default to "--exact".
Waiting for an Ack.
* jc/push-cert-hmac-optim (2014-09-25) 2 commits
- receive-pack: truncate hmac early and convert only necessary bytes
- sha1_to_hex: split out "hex-format n bytes" helper and use it
This is "we could do this if we wanted to", not "we measured and it
improves performance critical codepath".
Will perhaps drop.
* nd/multiple-work-trees (2014-09-27) 32 commits
. t2025: add a test to make sure grafts is working from a linked checkout
. checkout: don't require a work tree when checking out into a new one
. git_path(): keep "info/sparse-checkout" per work-tree
. count-objects: report unused files in $GIT_DIR/worktrees/...
. gc: support prune --worktrees
. gc: factor out gc.pruneexpire parsing code
. gc: style change -- no SP before closing parenthesis
. checkout: clean up half-prepared directories in --to mode
. checkout: reject if the branch is already checked out elsewhere
. prune: strategies for linked checkouts
. checkout: support checking out into a new working directory
. use new wrapper write_file() for simple file writing
. wrapper.c: wrapper to open a file, fprintf then close
. setup.c: support multi-checkout repo setup
. setup.c: detect $GIT_COMMON_DIR check_repository_format_gently()
. setup.c: convert check_repository_format_gently to use strbuf
. setup.c: detect $GIT_COMMON_DIR in is_git_directory()
. setup.c: convert is_git_directory() to use strbuf
. git-stash: avoid hardcoding $GIT_DIR/logs/....
. *.sh: avoid hardcoding $GIT_DIR/hooks/...
. git-sh-setup.sh: use rev-parse --git-path to get $GIT_DIR/objects
. $GIT_COMMON_DIR: a new environment variable
. commit: use SEQ_DIR instead of hardcoding "sequencer"
. fast-import: use git_path() for accessing .git dir instead of get_git_dir()
. reflog: avoid constructing .lock path with git_path
. *.sh: respect $GIT_INDEX_FILE
. git_path(): be aware of file relocation in $GIT_DIR
. path.c: group git_path(), git_pathdup() and strbuf_git_path() together
. path.c: rename vsnpath() to do_git_path()
. git_snpath(): retire and replace with strbuf_git_path()
. path.c: make get_pathname() call sites return const char *
. path.c: make get_pathname() return strbuf instead of static buffer
A replacement for contrib/workdir/git-new-workdir that does not
rely on symbolic links and make sharing of objects and refs safer
by making the borrowee and borrowers aware of each other.
A few tests need some tweaks for MinGW ($gmane/{257756,257757}).
Conflicts with rs/ref-transaction so ejected for now, waiting for a
reroll.
* mt/patch-id-stable (2014-06-10) 1 commit
- patch-id: change default to stable
Teaches "git patch-id" to compute the patch ID that does not change
when the files in a single patch is reordered. As this new algorithm
is backward incompatible, the last bit to flip it to be the default
is left out of 'master' for now.
Nobody seems to be jumping up & down requesting this last step,
which makes the result somewhat backward incompatible.
Will perhaps drop.
* tr/remerge-diff (2014-11-10) 9 commits
- t4213: avoid "|" in sed regexp
- log --remerge-diff: show what the conflict resolution changed
- name-hash: allow dir hashing even when !ignore_case
- merge-recursive: allow storing conflict hunks in index
- merge_diff_mode: fold all merge diff variants into an enum
- combine-diff: do not pass revs->dense_combined_merges redundantly
- merge-recursive: -Xindex-only to leave worktree unchanged
- merge-recursive: internal flag to avoid touching the worktree
- merge-recursive: remove dead conditional in update_stages()
"log -p" output learns a new way to let users inspect a merge
commit by showing the differences between the automerged result
with conflicts the person who recorded the merge would have seen
and the final conflict resolution that was recorded in the merge.
Waiting for a reroll ($gmane/256591).
* hv/submodule-config (2014-11-11) 4 commits
- do not die on error of parsing fetchrecursesubmodules option
- use new config API for worktree configurations of submodules
- extract functions for submodule config set and lookup
- implement submodule config cache for lookup of submodule names
Kicked back to 'pu' per request ($gmane/255610).
* jk/pack-bitmap (2014-08-04) 1 commit
- pack-bitmap: do not use gcc packed attribute
Hold, waiting for Karsten's replacement.
* ab/add-interactive-show-diff-func-name (2014-05-12) 2 commits
- SQUASH??? git-add--interactive: Preserve diff heading when splitting hunks
- git-add--interactive: Preserve diff heading when splitting hunks
Waiting for a reroll.
* jn/gitweb-utf8-in-links (2014-05-27) 1 commit
- gitweb: Harden UTF-8 handling in generated links
$gmane/250758?
* ss/userdiff-update-csharp-java (2014-06-02) 2 commits
- userdiff: support Java try keyword
- userdiff: support C# async methods and correct C# keywords
Reviews sent; waiting for a response.
* bg/rebase-off-of-previous-branch (2014-04-16) 1 commit
- git-rebase: print name of rev when using shorthand
Teach "git rebase -" to report the concrete name of the branch
(i.e. the previous one).
But it stops short and does not do the same for "git rebase @{-1}".
Expecting a reroll.
* rb/merge-prepare-commit-msg-hook (2014-01-10) 4 commits
- merge: drop unused arg from abort_commit method signature
- merge: make prepare_to_commit responsible for write_merge_state
- t7505: ensure cleanup after hook blocks merge
- t7505: add missing &&
Expose more merge states (e.g. $GIT_DIR/MERGE_MODE) to hooks that
run during "git merge". The log message stresses too much on one
hook, prepare-commit-msg, but it would equally apply to other hooks
like post-merge, I think.
Waiting for a reroll.
* jc/graph-post-root-gap (2013-12-30) 3 commits
- WIP: document what we want at the end
- graph: remove unused code a bit
- graph: stuff the current commit into graph->columns[]
This was primarily a RFH ($gmane/239580).
* tg/perf-lib-test-perf-cleanup (2013-09-19) 2 commits
- perf-lib: add test_perf_cleanup target
- perf-lib: split starting the test from the execution
Add test_perf_cleanup shell function to the perf suite, that allows
the script writers to define a test with a clean-up action.
Will hold.
* jc/show-branch (2014-03-24) 5 commits
- show-branch: use commit slab to represent bitflags of arbitrary width
- show-branch.c: remove "all_mask"
- show-branch.c: abstract out "flags" operation
- show-branch.c: lift all_mask/all_revs to a global static
- show-branch.c: update comment style
Waiting for the final step to lift the hard-limit before sending it out.
--------------------------------------------------
[Cooking]
* jk/approxidate-avoid-y-d-m-over-future-dates (2014-11-13) 2 commits
(merged to 'next' on 2014-11-14 at 3c11a1b)
+ approxidate: allow ISO-like dates far in the future
+ pass TIME_DATE_NOW to approxidate future-check
Traditionally we tried to avoid interpreting date strings given by
the user as future dates, e.g. GIT_COMMITTER_DATE=2014-12-10 when
used early November 2014 was taken as "October 12, 2014" because it
is likely that a date in the future, December 10, is a mistake.
Loosen this and do not tiebreak by future-ness of the date when
(1) ISO-like format is used, and
(2) the string can make sense interpreted as both y-m-d and y-d-m.
Will cook in 'next' throughout the remainder of the cycle.
* jk/checkout-from-tree (2014-11-13) 1 commit
(merged to 'next' on 2014-11-14 at ddbffb0)
+ checkout $tree: do not throw away unchanged index entries
Will merge to 'master'.
* mb/enable-lib-terminal-test-on-newer-darwin (2014-11-14) 1 commit
(merged to 'next' on 2014-11-14 at b2aae27)
+ t/lib-terminal: allow TTY tests to run under recent Mac OS
Will cook in 'next' throughout the remainder of the cycle.
We probably should drop this, though. ($gmane/259609).
* sv/get-builtin (2014-11-13) 1 commit
(merged to 'next' on 2014-11-14 at 9497e17)
+ builtin: move builtin retrieval to get_builtin()
Will merge to 'master'.
* br/imap-send-verbosity (2014-11-05) 1 commit
(merged to 'next' on 2014-11-12 at d9e58ec)
+ imap-send: use parse options API to determine verbosity
(this branch is used by br/imap-send-via-libcurl.)
Will cook in 'next' throughout the remainder of the cycle.
* br/imap-send-via-libcurl (2014-11-10) 1 commit
(merged to 'next' on 2014-11-12 at 5327ab4)
+ git-imap-send: use libcurl for implementation
(this branch uses br/imap-send-verbosity.)
Will cook in 'next' throughout the remainder of the cycle.
* cc/interpret-trailers-more (2014-11-10) 4 commits
- trailer: add test with an old style conflict block
- trailer: reuse ignore_non_trailer() to ignore conflict lines
- commit: make ignore_non_trailer() non static
- Merge branch 'jc/conflict-hint' into cc/interpret-trailers-more
(this branch uses jc/conflict-hint.)
* js/push-to-update (2014-11-13) 1 commit
- Add another option for receive.denyCurrentBranch
Still being discussed but we seem to have agreed what the desired
semantics should be.
* rs/env-array-in-child-process (2014-11-10) 1 commit
(merged to 'next' on 2014-11-14 at 3f6ba07)
+ use args member of struct child_process
Will merge to 'master'.
* tq/git-ssh-command (2014-11-10) 1 commit
(merged to 'next' on 2014-11-14 at 83f5dae)
+ git_connect: set ssh shell command in GIT_SSH_COMMAND
Allow passing extra set of arguments when ssh is invoked to create
an encrypted & authenticated connection, which is not possible with
existing GIT_SSH mechanism, which was designed more to match what
other programs with similar variables did, not necessarily to be
more useful.
Will merge to 'master'.
* ms/submodule-update-config-doc (2014-11-03) 1 commit
- submodule: clarify documentation for update subcommand
Needs a reroll ($gmane/259037).
* nd/lockfile-absolute (2014-11-03) 1 commit
(merged to 'next' on 2014-11-06 at 68722a9)
+ lockfile.c: store absolute path
The lockfile API can get confused which file to clean up when the
process moved the $cwd after creating a lockfile.
Will cook in 'next' throughout the remainder of the cycle.
* jh/empty-notes (2014-11-14) 9 commits
(merged to 'next' on 2014-11-18 at 9eeb338)
+ t3301: modernize style
+ notes: empty notes should be shown by 'git log'
+ builtin/notes: add --allow-empty, to allow storing empty notes
+ builtin/notes: split create_note() to clarify add vs. remove logic
+ builtin/notes: simplify early exit code in add()
+ builtin/notes: refactor note file path into struct note_data
+ builtin/notes: improve naming
+ t3301: verify that 'git notes' removes empty notes by default
+ builtin/notes: fix premature failure when trying to add the empty blob
A request to store an empty note via "git notes" meant to remove
note from the object but with --allow-empty we will store a (surprise!)
note that is empty. In the longer run, we might want to deprecate
the somewhat unintuitive "emptying means deletion" behaviour.
* jc/merge-bases (2014-10-30) 2 commits
(merged to 'next' on 2014-11-06 at 491e576)
+ get_merge_bases(): always clean-up object flags
+ bisect: clean flags after checking merge bases
Will cook in 'next' throughout the remainder of the cycle.
* jc/strbuf-add-lines-avoid-sp-ht-sequence (2014-10-27) 1 commit
(merged to 'next' on 2014-10-29 at 9167582)
+ strbuf_add_commented_lines(): avoid SP-HT sequence in commented lines
The commented output used to blindly add a SP before the payload
line, resulting in "# \t<indented text>\n" when the payload began
with a HT. Instead, produce "#\t<indented text>\n".
Will cook in 'next' throughout the remainder of the cycle.
* nd/untracked-cache (2014-10-27) 19 commits
- t7063: tests for untracked cache
- update-index: test the system before enabling untracked cache
- update-index: manually enable or disable untracked cache
- status: enable untracked cache
- untracked cache: mark index dirty if untracked cache is updated
- untracked cache: print stats with $GIT_TRACE_UNTRACKED_STATS
- untracked cache: avoid racy timestamps
- read-cache.c: split racy stat test to a separate function
- untracked cache: invalidate at index addition or removal
- untracked cache: load from UNTR index extension
- untracked cache: save to an index extension
- untracked cache: don't open non-existent .gitignore
- untracked cache: mark what dirs should be recursed/saved
- untracked cache: record/validate dir mtime and reuse cached output
- untracked cache: make a wrapper around {open,read,close}dir()
- untracked cache: invalidate dirs recursively if .gitignore changes
- untracked cache: initial untracked cache validation
- untracked cache: record .gitignore information and dir hierarchy
- dir.c: optionally compute sha-1 of a .gitignore file
* zk/grep-color-words (2014-10-27) 2 commits
(merged to 'next' on 2014-10-28 at 4d0457c)
+ Revert "grep: fix match highlighting for combined patterns with context lines"
(merged to 'next' on 2014-10-24 at 2d2f8f8)
+ grep: fix match highlighting for combined patterns with context lines
rs/grep-color-words topic solves it in a different way.
Will discard.
* jc/conflict-hint (2014-10-28) 4 commits
(merged to 'next' on 2014-10-29 at 693250f)
+ merge & sequencer: turn "Conflicts:" hint into a comment
+ builtin/commit.c: extract ignore_non_trailer() helper function
+ merge & sequencer: unify codepaths that write "Conflicts:" hint
+ builtin/merge.c: drop a parameter that is never used
(this branch is used by cc/interpret-trailers-more.)
Unlike all the other hints given in the commit log editor, the list
of conflicted paths were appended at the end without commented out.
Will cook in 'next' throughout the remainder of the cycle.
* jc/diff-b-m (2014-10-23) 1 commit
(merged to 'next' on 2014-10-28 at 4daedb1)
+ diff -B -M: fix output for "copy and then rewrite" case
Fix long-standing bug in "diff -B -M" output.
Will cook in 'next' throughout the remainder of the cycle.
* jc/checkout-local-track-report (2014-10-14) 1 commit
(merged to 'next' on 2014-10-21 at f636a00)
+ checkout: report upstream correctly even with loosely defined branch.*.merge
The report from "git checkout" on a branch that builds on another
local branch by setting its branch.*.merge to branch name (not a
full refname) incorrectly said that the upstream is gone.
Will cook in 'next' throughout the remainder of the cycle.
* jc/clone-borrow (2014-10-15) 1 commit
(merged to 'next' on 2014-10-21 at b76ea34)
+ clone: --dissociate option to mark that reference is only temporary
Allow "git clone --reference" to be used more safely.
Will cook in 'next' throughout the remainder of the cycle.
--------------------------------------------------
[Discarded]
* jt/timer-settime (2014-08-29) 6 commits
. use timer_settime() for new platforms
. autoconf: check for timer_settime()
. autoconf: check for struct itimerspec
. autoconf: check for struct sigevent
. autoconf: check for struct timespec
. autoconf: check for timer_t
Was wanting for a reroll.
* rs/ref-transaction-reflog (2014-11-03) 15 commits
. refs.c: allow deleting refs with a broken sha1
. refs.c: remove lock_any_ref_for_update
. refs.c: make unlock_ref/close_ref/commit_ref static
. refs.c: rename log_ref_setup to create_reflog
. reflog.c: use a reflog transaction when writing during expire
. refs.c: allow multiple reflog updates during a single transaction
. refs.c: only write reflog update if msg is non-NULL
. refs.c: add a flag to allow reflog updates to truncate the log
. refs.c: add a transaction function to append a reflog entry
. copy.c: make copy_fd preserve meaningful errno
. refs.c: add a function to append a reflog entry to a fd
. refs.c: add a new update_type field to ref_update
. refs.c: rename the transaction functions
. refs.c: make ref_transaction_delete a wrapper for ref_transaction_update
. refs.c make ref_transaction_create a wrapper to ref_transaction_update
(this branch is used by rs/ref-transaction-rename and rs/ref-transaction-send-pack.)
Stefan Beller started working on reorganizing these three series,
which unfortunately did not see much reviews.
* rs/ref-transaction-rename (2014-11-07) 16 commits
. refs.c: add an err argument to pack_refs
. refs.c: make lock_packed_refs take an err argument
. refs.c: make add_packed_ref return an error instead of calling die
. refs.c: replace the onerr argument in update_ref with a strbuf err
. refs.c: make the *_packed_refs functions static
. refs.c: make repack_without_refs static
. remote.c: use a transaction for deleting refs
. refs.c: write updates to packed refs when a transaction has more than one ref
. refs.c: move reflog updates into its own function
. refs.c: rollback the lockfile before we die() in repack_without_refs
. refs.c: update rename_ref to use a transaction
. refs.c: add transaction support for renaming a reflog
. refs.c: use a stringlist for repack_without_refs
. refs.c: use packed refs when deleting refs during a transaction
. refs.c: return error instead of dying when locking fails during transaction
. refs.c: allow passing raw git_committer_info as email to _update_reflog
(this branch is used by rs/ref-transaction-send-pack; uses rs/ref-transaction-reflog.)
Stefan Beller started working on reorganizing these three series,
which unfortunately did not see much reviews.
* rs/ref-transaction-send-pack (2014-11-07) 7 commits
. refs.c: add an err argument to create_symref
. refs.c: add an err argument to create_reflog
. t5543-atomic-push.sh: add basic tests for atomic pushes
. push.c: add an --atomic-push argument
. receive-pack.c: use a single transaction when atomic-push is negotiated
. send-pack.c: add an --atomic-push command line argument
. receive-pack.c: add protocol support to negotiate atomic-push
(this branch uses rs/ref-transaction-reflog and rs/ref-transaction-rename.)
Stefan Beller started working on reorganizing these three series,
which unfortunately did not see much reviews.
* sb/simplify-repack-without-refs (2014-11-20) 1 commit
. refs.c: use a string_list for repack_without_refs
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH] pack-bitmap: do not use gcc packed attribute
2014-11-26 23:09 What's cooking in git.git (Nov 2014, #04; Wed, 26) Junio C Hamano
@ 2014-11-27 5:24 ` Jeff King
2014-11-27 18:39 ` What's cooking in git.git (Nov 2014, #04; Wed, 26) brian m. carlson
1 sibling, 0 replies; 13+ messages in thread
From: Jeff King @ 2014-11-27 5:24 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Karsten Blees, git
On Wed, Nov 26, 2014 at 03:09:45PM -0800, Junio C Hamano wrote:
> * jk/pack-bitmap (2014-08-04) 1 commit
> - pack-bitmap: do not use gcc packed attribute
>
> Hold, waiting for Karsten's replacement.
I got tired of waiting, so here it is, I hope good enough for inclusion.
-- >8 --
From: Karsten Blees <blees@dcon.de>
Subject: pack-bitmap: do not use gcc packed attribute
The "__attribute__" flag may be a noop on some compilers.
That's OK as long as the code is correct without the
attribute, but in this case it is not. We would typically
end up with a struct that is 2 bytes too long due to struct
padding, breaking both reading and writing of bitmaps.
Instead of marshalling the data in a struct, let's just
provide helpers for reading and writing the appropriate
types. Besides being correct on all platforms, the result is
more efficient and simpler to read.
Signed-off-by: Karsten Blees <blees@dcon.de>
Signed-off-by: Jeff King <peff@peff.net>
---
From Karsten's original, the three changes I made (aside from the commit
message) were:
1. The _u32 helpers are now _be32, to make it clear that they are
dealing with big-endian integers (and it matches get/put_be32;
dropping the "u" is OK as it is implied by dealing with byte
ordering). I left the _u8 variants as-is; I do not think there is
precedent for a similar name for single bytes (and the "u" is
meaningful there). Technically you can accomplish the same thing
with a single call to sha1write, but I think the helper makes the
calling code flow better.
2. I moved the sha1write_* helpers into csum-file.h. It's possible
we will find other callers. I left the read_* variants as local
to pack-bitmap.c. In theory we could use them elsewhere, but I
could not find any other location that used the same "mmap base +
pos" pattern. Some similar code uses a simple pointer which is
updated, which would yield something like:
uint32_t read_be32(unsigned char **data)
{
uint32_t ret = get_be32(*data);
(*data) += sizeof(ret);
return ret;
}
In theory we could adapt the bitmap code here to use a similar
system, but it would involve a bit of surgery (we push the "pos"
pointer forward in a lot of places, not just here, and they would
all need to be converted). I don't think it's worth the trouble.
The original discussion also raised the question of whether we
could do a straight open/read on the bitmap file rather than
mmap-ing it. The answer is yes, though it would similarly involve a
lot of surgery. Moreover, it's possible that future versions of the
bitmap format would benefit from being mmap'd (this one does not).
So unless there is a compelling reason to switch away from mmap,
I think it makes sense to keep the code as-is.
3. I dropped casts from uint8_t to int in the assignment of
xor_offset, etc. These aren't doing anything (the compiler knows
both types and handles the conversion fine, and we know that a
uint8_t will always fit into an int on any sane platform).
csum-file.h | 11 +++++++++++
pack-bitmap-write.c | 8 +++-----
pack-bitmap.c | 22 +++++++++++++++-------
pack-bitmap.h | 6 ------
4 files changed, 29 insertions(+), 18 deletions(-)
diff --git a/csum-file.h b/csum-file.h
index bb543d5..7530927 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -39,4 +39,15 @@ extern void sha1flush(struct sha1file *f);
extern void crc32_begin(struct sha1file *);
extern uint32_t crc32_end(struct sha1file *);
+static inline void sha1write_u8(struct sha1file *f, uint8_t data)
+{
+ sha1write(f, &data, sizeof(data));
+}
+
+static inline void sha1write_be32(struct sha1file *f, uint32_t data)
+{
+ data = htonl(data);
+ sha1write(f, &data, sizeof(data));
+}
+
#endif
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 8029ae3..c05d138 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -472,7 +472,6 @@ static void write_selected_commits_v1(struct sha1file *f,
for (i = 0; i < writer.selected_nr; ++i) {
struct bitmapped_commit *stored = &writer.selected[i];
- struct bitmap_disk_entry on_disk;
int commit_pos =
sha1_pos(stored->commit->object.sha1, index, index_nr, sha1_access);
@@ -480,11 +479,10 @@ static void write_selected_commits_v1(struct sha1file *f,
if (commit_pos < 0)
die("BUG: trying to write commit not in index");
- on_disk.object_pos = htonl(commit_pos);
- on_disk.xor_offset = stored->xor_offset;
- on_disk.flags = stored->flags;
+ sha1write_be32(f, commit_pos);
+ sha1write_u8(f, stored->xor_offset);
+ sha1write_u8(f, stored->flags);
- sha1write(f, &on_disk, sizeof(on_disk));
dump_bitmap(f, stored->write_as);
}
}
diff --git a/pack-bitmap.c b/pack-bitmap.c
index a1f3c0d..6a81841 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -197,13 +197,24 @@ static struct stored_bitmap *store_bitmap(struct bitmap_index *index,
return stored;
}
+static inline uint32_t read_be32(const unsigned char *buffer, size_t *pos)
+{
+ uint32_t result = get_be32(buffer + *pos);
+ (*pos) += sizeof(result);
+ return result;
+}
+
+static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)
+{
+ return buffer[(*pos)++];
+}
+
static int load_bitmap_entries_v1(struct bitmap_index *index)
{
static const size_t MAX_XOR_OFFSET = 160;
uint32_t i;
struct stored_bitmap **recent_bitmaps;
- struct bitmap_disk_entry *entry;
recent_bitmaps = xcalloc(MAX_XOR_OFFSET, sizeof(struct stored_bitmap));
@@ -214,15 +225,12 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)
uint32_t commit_idx_pos;
const unsigned char *sha1;
- entry = (struct bitmap_disk_entry *)(index->map + index->map_pos);
- index->map_pos += sizeof(struct bitmap_disk_entry);
+ commit_idx_pos = read_be32(index->map, &index->map_pos);
+ xor_offset = read_u8(index->map, &index->map_pos);
+ flags = read_u8(index->map, &index->map_pos);
- commit_idx_pos = ntohl(entry->object_pos);
sha1 = nth_packed_object_sha1(index->pack, commit_idx_pos);
- xor_offset = (int)entry->xor_offset;
- flags = (int)entry->flags;
-
bitmap = read_bitmap_1(index);
if (!bitmap)
return -1;
diff --git a/pack-bitmap.h b/pack-bitmap.h
index 8b7f4e9..487600b 100644
--- a/pack-bitmap.h
+++ b/pack-bitmap.h
@@ -5,12 +5,6 @@
#include "khash.h"
#include "pack-objects.h"
-struct bitmap_disk_entry {
- uint32_t object_pos;
- uint8_t xor_offset;
- uint8_t flags;
-} __attribute__((packed));
-
struct bitmap_disk_header {
char magic[4];
uint16_t version;
--
2.2.0.rc2.402.g4519813
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: What's cooking in git.git (Nov 2014, #04; Wed, 26)
2014-11-26 23:09 What's cooking in git.git (Nov 2014, #04; Wed, 26) Junio C Hamano
2014-11-27 5:24 ` [PATCH] pack-bitmap: do not use gcc packed attribute Jeff King
@ 2014-11-27 18:39 ` brian m. carlson
2014-11-30 8:35 ` Duy Nguyen
2014-12-01 2:09 ` Junio C Hamano
1 sibling, 2 replies; 13+ messages in thread
From: brian m. carlson @ 2014-11-27 18:39 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
[-- Attachment #1: Type: text/plain, Size: 1540 bytes --]
On Wed, Nov 26, 2014 at 03:09:45PM -0800, Junio C Hamano wrote:
> * nd/untracked-cache (2014-10-27) 19 commits
> - t7063: tests for untracked cache
> - update-index: test the system before enabling untracked cache
> - update-index: manually enable or disable untracked cache
> - status: enable untracked cache
> - untracked cache: mark index dirty if untracked cache is updated
> - untracked cache: print stats with $GIT_TRACE_UNTRACKED_STATS
> - untracked cache: avoid racy timestamps
> - read-cache.c: split racy stat test to a separate function
> - untracked cache: invalidate at index addition or removal
> - untracked cache: load from UNTR index extension
> - untracked cache: save to an index extension
> - untracked cache: don't open non-existent .gitignore
> - untracked cache: mark what dirs should be recursed/saved
> - untracked cache: record/validate dir mtime and reuse cached output
> - untracked cache: make a wrapper around {open,read,close}dir()
> - untracked cache: invalidate dirs recursively if .gitignore changes
> - untracked cache: initial untracked cache validation
> - untracked cache: record .gitignore information and dir hierarchy
> - dir.c: optionally compute sha-1 of a .gitignore file
You didn't comment on the status of this branch, and I'm interested.
--
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: What's cooking in git.git (Nov 2014, #04; Wed, 26)
2014-11-27 18:39 ` What's cooking in git.git (Nov 2014, #04; Wed, 26) brian m. carlson
@ 2014-11-30 8:35 ` Duy Nguyen
2014-11-30 18:18 ` brian m. carlson
2014-12-01 2:09 ` Junio C Hamano
1 sibling, 1 reply; 13+ messages in thread
From: Duy Nguyen @ 2014-11-30 8:35 UTC (permalink / raw)
To: brian m. carlson; +Cc: Junio C Hamano, Git Mailing List
On Fri, Nov 28, 2014 at 1:39 AM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> On Wed, Nov 26, 2014 at 03:09:45PM -0800, Junio C Hamano wrote:
>> * nd/untracked-cache (2014-10-27) 19 commits
>> - t7063: tests for untracked cache
>> - update-index: test the system before enabling untracked cache
>> - update-index: manually enable or disable untracked cache
>> - status: enable untracked cache
>> - untracked cache: mark index dirty if untracked cache is updated
>> - untracked cache: print stats with $GIT_TRACE_UNTRACKED_STATS
>> - untracked cache: avoid racy timestamps
>> - read-cache.c: split racy stat test to a separate function
>> - untracked cache: invalidate at index addition or removal
>> - untracked cache: load from UNTR index extension
>> - untracked cache: save to an index extension
>> - untracked cache: don't open non-existent .gitignore
>> - untracked cache: mark what dirs should be recursed/saved
>> - untracked cache: record/validate dir mtime and reuse cached output
>> - untracked cache: make a wrapper around {open,read,close}dir()
>> - untracked cache: invalidate dirs recursively if .gitignore changes
>> - untracked cache: initial untracked cache validation
>> - untracked cache: record .gitignore information and dir hierarchy
>> - dir.c: optionally compute sha-1 of a .gitignore file
>
> You didn't comment on the status of this branch, and I'm interested.
I'm not Junio :) but I think the core changes are done. I wanted to
actually add watchman support on top of untracked cache as well to see
if it needs any more changes. I think I can see how it could be done
now (not easy, but not terribly hard). I'm going to resend soon to fix
some minor bugs (in a reroll that Junio has not picked up) and change
file format to be more compact.
--
Duy
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: What's cooking in git.git (Nov 2014, #04; Wed, 26)
2014-11-30 8:35 ` Duy Nguyen
@ 2014-11-30 18:18 ` brian m. carlson
0 siblings, 0 replies; 13+ messages in thread
From: brian m. carlson @ 2014-11-30 18:18 UTC (permalink / raw)
To: Duy Nguyen; +Cc: Junio C Hamano, Git Mailing List
[-- Attachment #1: Type: text/plain, Size: 853 bytes --]
On Sun, Nov 30, 2014 at 03:35:33PM +0700, Duy Nguyen wrote:
> I'm not Junio :) but I think the core changes are done. I wanted to
> actually add watchman support on top of untracked cache as well to see
> if it needs any more changes. I think I can see how it could be done
> now (not easy, but not terribly hard). I'm going to resend soon to fix
> some minor bugs (in a reroll that Junio has not picked up) and change
> file format to be more compact.
Excellent. I want to see if this improves performance over sshfs, but I
don't want to enable it until I can put it on the server machine as well
to prevent incompatibilities.
--
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: What's cooking in git.git (Nov 2014, #04; Wed, 26)
2014-11-27 18:39 ` What's cooking in git.git (Nov 2014, #04; Wed, 26) brian m. carlson
2014-11-30 8:35 ` Duy Nguyen
@ 2014-12-01 2:09 ` Junio C Hamano
1 sibling, 0 replies; 13+ messages in thread
From: Junio C Hamano @ 2014-12-01 2:09 UTC (permalink / raw)
To: brian m. carlson; +Cc: git
"brian m. carlson" <sandals@crustytoothpaste.net> writes:
> On Wed, Nov 26, 2014 at 03:09:45PM -0800, Junio C Hamano wrote:
>> * nd/untracked-cache (2014-10-27) 19 commits
>> ...
>> - dir.c: optionally compute sha-1 of a .gitignore file
>
> You didn't comment on the status of this branch, and I'm interested.
I think we already saw a few comments responded by the author with
"thanks, will include in a reroll", and that is why the listed are
not even the latest round.
The status was "waiting for a reroll". I haven't caught up with the
traffic for the past few days, though.
^ permalink raw reply [flat|nested] 13+ messages in thread
* struct hashmap_entry packing
@ 2014-07-28 17:17 Jeff King
2014-07-29 20:40 ` Karsten Blees
0 siblings, 1 reply; 13+ messages in thread
From: Jeff King @ 2014-07-28 17:17 UTC (permalink / raw)
To: Karsten Blees; +Cc: git
Hi Karsten,
The hashmap_entry documentation claims:
`struct hashmap_entry`::
An opaque structure representing an entry in the hash table,
which must be used as first member of user data structures.
Ideally it should be followed by an int-sized member to prevent
unused memory on 64-bit systems due to alignment.
I'm not sure if the statement about alignment is true. If I have a
struct like:
struct magic {
struct hashmap_entry map;
int x;
};
the statement above implies that I should be able to fit this into only
16 bytes on an LP64 system. But I can't convince gcc to do it. And I
think that makes sense, if you consider code like:
memset(&magic.map, 0, sizeof(struct hashmap_entry));
The sizeof() has to be the same regardless of whether the hashmap_entry
is standalone or in another struct, and therefore must be padded up to
16 bytes. If we stored "x" in that padding in the combined struct, it
would be overwritten by our memset.
Am I missing anything? If this is the case, we should probably drop that
bit from the documentation. It's possible that we could get around it by
embedding the hashmap_entry elements directly into the parent struct,
but we would be counting on a reader dereferencing it as a hashmap_entry
seeing the members at the exact same offset. I'd imagine that's one of
those things that holds most of the time, but is violating the standard.
It's probably not worth it to save a few bytes.
-Peff
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: struct hashmap_entry packing
2014-07-28 17:17 struct hashmap_entry packing Jeff King
@ 2014-07-29 20:40 ` Karsten Blees
2014-08-01 22:37 ` Jeff King
0 siblings, 1 reply; 13+ messages in thread
From: Karsten Blees @ 2014-07-29 20:40 UTC (permalink / raw)
To: Jeff King; +Cc: git
Am 28.07.2014 19:17, schrieb Jeff King:
> Hi Karsten,
>
> The hashmap_entry documentation claims:
>
> `struct hashmap_entry`::
>
> An opaque structure representing an entry in the hash table,
> which must be used as first member of user data structures.
> Ideally it should be followed by an int-sized member to prevent
> unused memory on 64-bit systems due to alignment.
>
> I'm not sure if the statement about alignment is true. If I have a
> struct like:
>
> struct magic {
> struct hashmap_entry map;
> int x;
> };
>
> the statement above implies that I should be able to fit this into only
> 16 bytes on an LP64 system. But I can't convince gcc to do it. And I
> think that makes sense, if you consider code like:
>
> memset(&magic.map, 0, sizeof(struct hashmap_entry));
>
> The sizeof() has to be the same regardless of whether the hashmap_entry
> is standalone or in another struct, and therefore must be padded up to
> 16 bytes. If we stored "x" in that padding in the combined struct, it
> would be overwritten by our memset.
>
The struct-packing patch was ultimately dropped because there was no way
to reliably make it work on all platforms. See [1] for discussion, [2] for
the final, 'most compatible' version.
> Am I missing anything? If this is the case, we should probably drop that
> bit from the documentation.
Hmmm. Now that we have "__attribute__((packed))" in pack-bitmap.h, perhaps
we should do the same for stuct hashmap_entry? (Which was the original
proposal anyway...). Only works for GCC, but that should cover most builds
/ platforms.
Btw.: Using struct-packing on 'struct bitmap_disk_entry' means that the
binary format of .bitmap files is incompatible between GCC and other
builds, correct?
> It's possible that we could get around it by
> embedding the hashmap_entry elements directly into the parent struct,
Already tried that, see [3].
[1] http://article.gmane.org/gmane.comp.version-control.git/239069
[2] http://article.gmane.org/gmane.comp.version-control.git/241865
[3] http://article.gmane.org/gmane.comp.version-control.git/239435
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: struct hashmap_entry packing
2014-07-29 20:40 ` Karsten Blees
@ 2014-08-01 22:37 ` Jeff King
2014-08-01 23:10 ` [PATCH] pack-bitmap: do not use gcc packed attribute Jeff King
0 siblings, 1 reply; 13+ messages in thread
From: Jeff King @ 2014-08-01 22:37 UTC (permalink / raw)
To: Karsten Blees; +Cc: git
On Tue, Jul 29, 2014 at 10:40:12PM +0200, Karsten Blees wrote:
> > The sizeof() has to be the same regardless of whether the hashmap_entry
> > is standalone or in another struct, and therefore must be padded up to
> > 16 bytes. If we stored "x" in that padding in the combined struct, it
> > would be overwritten by our memset.
> >
>
> The struct-packing patch was ultimately dropped because there was no way
> to reliably make it work on all platforms. See [1] for discussion, [2] for
> the final, 'most compatible' version.
Thanks for the pointers; I should have guessed there was more to it and
searched the archive myself.
> Hmmm. Now that we have "__attribute__((packed))" in pack-bitmap.h, perhaps
> we should do the same for stuct hashmap_entry? (Which was the original
> proposal anyway...). Only works for GCC, but that should cover most builds
> / platforms.
I don't see any reason to avoid the packed attribute, if it helps us. As
you noted, anything using __attribute__ probably supports it, and if
not, we can conditionally #define PACKED_STRUCT or something, like we do
for NORETURN. Since it's purely an optimization, if another compiler
doesn't use it, no big deal.
That being said, I don't know if those padding bytes are actually
causing a measurable slowdown. It may not even be worth the trouble.
> Btw.: Using struct-packing on 'struct bitmap_disk_entry' means that the
> binary format of .bitmap files is incompatible between GCC and other
> builds, correct?
The on-disk format is defined by JGit; if there are differences between
the builds, that's a bug (and I would not be too surprised if there is
one, as bitmaps have gotten very extensive testing on 32- and 64-bit
gcc, but probably not much elsewhere).
We do use structs to represent disk structures in other bits of the
packfile code (e.g., struct pack_idx_header), but the struct is vanilla
enough that we assume every compiler gives us two tightly-packed 32-bit
integers without having to bother with the "packed" attribute (and it
seems to have worked in practice).
We should probably be more careful with that bitmap code. It looks like
it wouldn't be too bad to drop it. I'll see if I can come up with a
patch.
-Peff
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH] pack-bitmap: do not use gcc packed attribute
2014-08-01 22:37 ` Jeff King
@ 2014-08-01 23:10 ` Jeff King
2014-08-01 23:12 ` Jeff King
2014-08-04 19:19 ` Karsten Blees
0 siblings, 2 replies; 13+ messages in thread
From: Jeff King @ 2014-08-01 23:10 UTC (permalink / raw)
To: Karsten Blees; +Cc: Junio C Hamano, git
On Fri, Aug 01, 2014 at 06:37:39PM -0400, Jeff King wrote:
> > Btw.: Using struct-packing on 'struct bitmap_disk_entry' means that the
> > binary format of .bitmap files is incompatible between GCC and other
> > builds, correct?
>
> The on-disk format is defined by JGit; if there are differences between
> the builds, that's a bug (and I would not be too surprised if there is
> one, as bitmaps have gotten very extensive testing on 32- and 64-bit
> gcc, but probably not much elsewhere).
>
> We do use structs to represent disk structures in other bits of the
> packfile code (e.g., struct pack_idx_header), but the struct is vanilla
> enough that we assume every compiler gives us two tightly-packed 32-bit
> integers without having to bother with the "packed" attribute (and it
> seems to have worked in practice).
>
> We should probably be more careful with that bitmap code. It looks like
> it wouldn't be too bad to drop it. I'll see if I can come up with a
> patch.
I confirmed that this does break horribly without the packed attribute
(as you'd expect; it's asking for 48-bit alignment!). p5310 notices it,
_if_ you have jgit installed to check against.
Here's a fix.
-- >8 --
Subject: pack-bitmap: do not use gcc packed attribute
The "__attribute__" flag may be a noop on some compilers.
That's OK as long as the code is correct without the
attribute, but in this case it is not. We would typically
end up with a struct that is 2 bytes too long due to struct
padding, breaking both reading and writing of bitmaps.
We can work around this by using an array of unsigned char
to represent the data, and relying on get/put_be32 to handle
alignment issues as we interact with the array.
Signed-off-by: Jeff King <peff@peff.net>
---
The accessors may be overkill; each function is called only a single
time in the whole codebase. But doing it this way rather than accessing
entry[4] inline at least puts the magic constants all in one place.
pack-bitmap-write.c | 10 ++++------
pack-bitmap.c | 12 ++++++------
pack-bitmap.h | 42 +++++++++++++++++++++++++++++++++++++-----
3 files changed, 47 insertions(+), 17 deletions(-)
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 5f1791a..f885a7a 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -473,7 +473,7 @@ static void write_selected_commits_v1(struct sha1file *f,
for (i = 0; i < writer.selected_nr; ++i) {
struct bitmapped_commit *stored = &writer.selected[i];
- struct bitmap_disk_entry on_disk;
+ unsigned char on_disk[BITMAP_DISK_ENTRY_LEN];
int commit_pos =
sha1_pos(stored->commit->object.sha1, index, index_nr, sha1_access);
@@ -481,11 +481,9 @@ static void write_selected_commits_v1(struct sha1file *f,
if (commit_pos < 0)
die("BUG: trying to write commit not in index");
- on_disk.object_pos = htonl(commit_pos);
- on_disk.xor_offset = stored->xor_offset;
- on_disk.flags = stored->flags;
-
- sha1write(f, &on_disk, sizeof(on_disk));
+ bitmap_disk_entry_create(on_disk, commit_pos,
+ stored->xor_offset, stored->flags);
+ sha1write(f, on_disk, sizeof(on_disk));
dump_bitmap(f, stored->write_as);
}
}
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 91e4101..1b2a473 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -203,7 +203,7 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)
uint32_t i;
struct stored_bitmap **recent_bitmaps;
- struct bitmap_disk_entry *entry;
+ unsigned char *entry;
recent_bitmaps = xcalloc(MAX_XOR_OFFSET, sizeof(struct stored_bitmap));
@@ -214,14 +214,14 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)
uint32_t commit_idx_pos;
const unsigned char *sha1;
- entry = (struct bitmap_disk_entry *)(index->map + index->map_pos);
- index->map_pos += sizeof(struct bitmap_disk_entry);
+ entry = index->map + index->map_pos;
+ index->map_pos += BITMAP_DISK_ENTRY_LEN;
- commit_idx_pos = ntohl(entry->object_pos);
+ commit_idx_pos = bitmap_disk_entry_object_pos(entry);
sha1 = nth_packed_object_sha1(index->pack, commit_idx_pos);
- xor_offset = (int)entry->xor_offset;
- flags = (int)entry->flags;
+ xor_offset = (int)bitmap_disk_entry_xor_offset(entry);
+ flags = (int)bitmap_disk_entry_flags(entry);
bitmap = read_bitmap_1(index);
if (!bitmap)
diff --git a/pack-bitmap.h b/pack-bitmap.h
index 8b7f4e9..0d57706 100644
--- a/pack-bitmap.h
+++ b/pack-bitmap.h
@@ -5,11 +5,43 @@
#include "khash.h"
#include "pack-objects.h"
-struct bitmap_disk_entry {
- uint32_t object_pos;
- uint8_t xor_offset;
- uint8_t flags;
-} __attribute__((packed));
+/*
+ * This is the equivalent of:
+ *
+ * uint32_t object_pos;
+ * uint8_t xor_offset;
+ * uint8_t flags;
+ *
+ * but due to the funny sizing, we cannot rely on the compiler to give us the
+ * exact struct packing we want. So let's treat it as an array and just provide
+ * a few helpers for accessing the components.
+ */
+#define BITMAP_DISK_ENTRY_LEN 6
+
+static inline void bitmap_disk_entry_create(unsigned char *on_disk,
+ uint32_t object_pos,
+ uint8_t xor_offset,
+ uint8_t flags)
+{
+ put_be32(on_disk, object_pos);
+ on_disk[4] = xor_offset;
+ on_disk[5] = flags;
+}
+
+static inline uint32_t bitmap_disk_entry_object_pos(unsigned char *on_disk)
+{
+ return get_be32(on_disk);
+}
+
+static inline uint8_t bitmap_disk_entry_xor_offset(unsigned char *on_disk)
+{
+ return on_disk[4];
+}
+
+static inline uint8_t bitmap_disk_entry_flags(unsigned char *on_disk)
+{
+ return on_disk[5];
+}
struct bitmap_disk_header {
char magic[4];
--
2.1.0.rc0.286.g5c67d74
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH] pack-bitmap: do not use gcc packed attribute
2014-08-01 23:10 ` [PATCH] pack-bitmap: do not use gcc packed attribute Jeff King
@ 2014-08-01 23:12 ` Jeff King
2014-08-04 19:19 ` Karsten Blees
1 sibling, 0 replies; 13+ messages in thread
From: Jeff King @ 2014-08-01 23:12 UTC (permalink / raw)
To: Karsten Blees; +Cc: Junio C Hamano, git
On Fri, Aug 01, 2014 at 07:10:44PM -0400, Jeff King wrote:
> I confirmed that this does break horribly without the packed attribute
> (as you'd expect; it's asking for 48-bit alignment!). p5310 notices it,
> _if_ you have jgit installed to check against.
Er, that should be t5310, of course. p5310 is the perf test, which does
not notice the problem. :)
-Peff
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] pack-bitmap: do not use gcc packed attribute
2014-08-01 23:10 ` [PATCH] pack-bitmap: do not use gcc packed attribute Jeff King
2014-08-01 23:12 ` Jeff King
@ 2014-08-04 19:19 ` Karsten Blees
2014-08-05 18:38 ` Vicent Martí
2014-08-05 18:47 ` Jeff King
1 sibling, 2 replies; 13+ messages in thread
From: Karsten Blees @ 2014-08-04 19:19 UTC (permalink / raw)
To: Jeff King; +Cc: Junio C Hamano, git
Am 02.08.2014 01:10, schrieb Jeff King:
> On Fri, Aug 01, 2014 at 06:37:39PM -0400, Jeff King wrote:
>
>>> Btw.: Using struct-packing on 'struct bitmap_disk_entry' means that the
>>> binary format of .bitmap files is incompatible between GCC and other
>>> builds, correct?
>>
>> The on-disk format is defined by JGit; if there are differences between
>> the builds, that's a bug (and I would not be too surprised if there is
>> one, as bitmaps have gotten very extensive testing on 32- and 64-bit
>> gcc, but probably not much elsewhere).
>>
>> We do use structs to represent disk structures in other bits of the
>> packfile code (e.g., struct pack_idx_header), but the struct is vanilla
>> enough that we assume every compiler gives us two tightly-packed 32-bit
>> integers without having to bother with the "packed" attribute (and it
>> seems to have worked in practice).
>>
>> We should probably be more careful with that bitmap code. It looks like
>> it wouldn't be too bad to drop it. I'll see if I can come up with a
>> patch.
>
> I confirmed that this does break horribly without the packed attribute
> (as you'd expect; it's asking for 48-bit alignment!). p5310 notices it,
> _if_ you have jgit installed to check against.
>
> Here's a fix.
>
> Subject: pack-bitmap: do not use gcc packed attribute
>
> The "__attribute__" flag may be a noop on some compilers.
> That's OK as long as the code is correct without the
> attribute, but in this case it is not. We would typically
> end up with a struct that is 2 bytes too long due to struct
> padding, breaking both reading and writing of bitmaps.
>
> We can work around this by using an array of unsigned char
> to represent the data, and relying on get/put_be32 to handle
> alignment issues as we interact with the array.
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
> The accessors may be overkill; each function is called only a single
> time in the whole codebase. But doing it this way rather than accessing
> entry[4] inline at least puts the magic constants all in one place.
>
[...]
Hmm, I wonder if it wouldn't be simpler to read / write the desired on-disk
structure directly, without copying to a uchar[6] first.
When writing, sha1write already buffers the data, so calling this with 4/1/1
bytes of payload shouldn't affect performance.
Similarly for reading - we already have a function to read a bitmap and
advance the 'file' position, why not have similar functions to read uint8/32
in a stream-based fashion?
This raises the question why we read via mmap at all (without munmap()ing the
file when done...). We copy all data into internal data structures anyway. Is
an fopen/fread-based solution (with fread_u8/_u32 helpers) that much slower?
Here's what I came up with (just a sketch, commit message is lacky and the
helper functions deserve a better place / name):
----8<-----
Subject: [PATCH] pack-bitmap: do not use packed structs to read / write bitmap
files
Signed-off-by: Karsten Blees <blees@dcon.de>
---
pack-bitmap-write.c | 18 +++++++++++++-----
pack-bitmap.c | 21 ++++++++++++++-------
pack-bitmap.h | 6 ------
3 files changed, 27 insertions(+), 18 deletions(-)
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 5f1791a..01995cb 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -465,6 +465,16 @@ static const unsigned char *sha1_access(size_t pos, void *table)
return index[pos]->sha1;
}
+static inline void sha1write_u8(struct sha1file *f, uint8_t data)
+{
+ sha1write(f, &data, sizeof(data));
+}
+static inline void sha1write_u32(struct sha1file *f, uint32_t data)
+{
+ data = htonl(data);
+ sha1write(f, &data, sizeof(data));
+}
+
static void write_selected_commits_v1(struct sha1file *f,
struct pack_idx_entry **index,
uint32_t index_nr)
@@ -473,7 +483,6 @@ static void write_selected_commits_v1(struct sha1file *f,
for (i = 0; i < writer.selected_nr; ++i) {
struct bitmapped_commit *stored = &writer.selected[i];
- struct bitmap_disk_entry on_disk;
int commit_pos =
sha1_pos(stored->commit->object.sha1, index, index_nr, sha1_access);
@@ -481,11 +490,10 @@ static void write_selected_commits_v1(struct sha1file *f,
if (commit_pos < 0)
die("BUG: trying to write commit not in index");
- on_disk.object_pos = htonl(commit_pos);
- on_disk.xor_offset = stored->xor_offset;
- on_disk.flags = stored->flags;
+ sha1write_u32(f, commit_pos);
+ sha1write_u8(f, stored->xor_offset);
+ sha1write_u8(f, stored->flags);
- sha1write(f, &on_disk, sizeof(on_disk));
dump_bitmap(f, stored->write_as);
}
}
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 91e4101..cb1b2dd 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -197,13 +197,23 @@ static struct stored_bitmap *store_bitmap(struct bitmap_index *index,
return stored;
}
+static inline uint32_t read_u32(const unsigned char *buffer, size_t *pos)
+{
+ uint32_t result = get_be32(buffer + *pos);
+ (*pos) += sizeof(result);
+ return result;
+}
+static inline uint8_t read_u8(const unsigned char *buffer, size_t *pos)
+{
+ return buffer[(*pos)++];
+}
+
static int load_bitmap_entries_v1(struct bitmap_index *index)
{
static const size_t MAX_XOR_OFFSET = 160;
uint32_t i;
struct stored_bitmap **recent_bitmaps;
- struct bitmap_disk_entry *entry;
recent_bitmaps = xcalloc(MAX_XOR_OFFSET, sizeof(struct stored_bitmap));
@@ -214,15 +224,12 @@ static int load_bitmap_entries_v1(struct bitmap_index *index)
uint32_t commit_idx_pos;
const unsigned char *sha1;
- entry = (struct bitmap_disk_entry *)(index->map + index->map_pos);
- index->map_pos += sizeof(struct bitmap_disk_entry);
+ commit_idx_pos = read_u32(index->map, &index->map_pos);
+ xor_offset = (int) read_u8(index->map, &index->map_pos);
+ flags = (int) read_u8(index->map, &index->map_pos);
- commit_idx_pos = ntohl(entry->object_pos);
sha1 = nth_packed_object_sha1(index->pack, commit_idx_pos);
- xor_offset = (int)entry->xor_offset;
- flags = (int)entry->flags;
-
bitmap = read_bitmap_1(index);
if (!bitmap)
return -1;
diff --git a/pack-bitmap.h b/pack-bitmap.h
index 8b7f4e9..487600b 100644
--- a/pack-bitmap.h
+++ b/pack-bitmap.h
@@ -5,12 +5,6 @@
#include "khash.h"
#include "pack-objects.h"
-struct bitmap_disk_entry {
- uint32_t object_pos;
- uint8_t xor_offset;
- uint8_t flags;
-} __attribute__((packed));
-
struct bitmap_disk_header {
char magic[4];
uint16_t version;
--
2.0.3.920.g16a4828.dirty
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH] pack-bitmap: do not use gcc packed attribute
2014-08-04 19:19 ` Karsten Blees
@ 2014-08-05 18:38 ` Vicent Martí
2014-08-05 18:47 ` Jeff King
1 sibling, 0 replies; 13+ messages in thread
From: Vicent Martí @ 2014-08-05 18:38 UTC (permalink / raw)
To: Karsten Blees; +Cc: Jeff King, Junio C Hamano, git
On Mon, Aug 4, 2014 at 9:19 PM, Karsten Blees <karsten.blees@gmail.com> wrote:
> This raises the question why we read via mmap at all
The first version of the pack bitmap format I wrote for GitHub was 50%
faster to load than this one because it was designed to be mmapable.
Eventually we moved to the JGit-compatible bitmap format (because I
get paid a lot of money to do as I'm told -- not because of some
inherent benefit of the JGit format), which needs to be read
sequentially, but I never bothered to change the mmap reading code.
I believe your patch makes a lot of sense -- at this point we could as
well remove the mmaping altogether and read the file sequentially.
Cheers,
vmg
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] pack-bitmap: do not use gcc packed attribute
2014-08-04 19:19 ` Karsten Blees
2014-08-05 18:38 ` Vicent Martí
@ 2014-08-05 18:47 ` Jeff King
2014-08-06 18:58 ` Karsten Blees
1 sibling, 1 reply; 13+ messages in thread
From: Jeff King @ 2014-08-05 18:47 UTC (permalink / raw)
To: Karsten Blees; +Cc: Junio C Hamano, git
On Mon, Aug 04, 2014 at 09:19:46PM +0200, Karsten Blees wrote:
> Hmm, I wonder if it wouldn't be simpler to read / write the desired on-disk
> structure directly, without copying to a uchar[6] first.
Probably. My initial attempt was to keep together the read/write logic
about which sizes each item is, but I think the result ended up
unnecessarily verbose and harder to follow.
> Here's what I came up with (just a sketch, commit message is lacky and the
> helper functions deserve a better place / name):
I like it. Want to clean it up and submit in place of mine?
-Peff
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] pack-bitmap: do not use gcc packed attribute
2014-08-05 18:47 ` Jeff King
@ 2014-08-06 18:58 ` Karsten Blees
2014-08-06 19:32 ` Junio C Hamano
0 siblings, 1 reply; 13+ messages in thread
From: Karsten Blees @ 2014-08-06 18:58 UTC (permalink / raw)
To: Jeff King; +Cc: Junio C Hamano, git
Am 05.08.2014 20:47, schrieb Jeff King:
> On Mon, Aug 04, 2014 at 09:19:46PM +0200, Karsten Blees wrote:
>
>> Hmm, I wonder if it wouldn't be simpler to read / write the desired on-disk
>> structure directly, without copying to a uchar[6] first.
>
> Probably. My initial attempt was to keep together the read/write logic
> about which sizes each item is, but I think the result ended up
> unnecessarily verbose and harder to follow.
>
Yeah, having read / write logic in different files is confusing, esp. when
not using structs to define the file format.
>> Here's what I came up with (just a sketch, commit message is lacky and the
>> helper functions deserve a better place / name):
>
> I like it. Want to clean it up and submit in place of mine?
>
Will do, but it will have to wait till next week.
> -Peff
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] pack-bitmap: do not use gcc packed attribute
2014-08-06 18:58 ` Karsten Blees
@ 2014-08-06 19:32 ` Junio C Hamano
0 siblings, 0 replies; 13+ messages in thread
From: Junio C Hamano @ 2014-08-06 19:32 UTC (permalink / raw)
To: Karsten Blees; +Cc: Jeff King, git
Karsten Blees <karsten.blees@gmail.com> writes:
> Am 05.08.2014 20:47, schrieb Jeff King:
>> On Mon, Aug 04, 2014 at 09:19:46PM +0200, Karsten Blees wrote:
>>
>>> Hmm, I wonder if it wouldn't be simpler to read / write the desired on-disk
>>> structure directly, without copying to a uchar[6] first.
>>
>> Probably. My initial attempt was to keep together the read/write logic
>> about which sizes each item is, but I think the result ended up
>> unnecessarily verbose and harder to follow.
>>
>
> Yeah, having read / write logic in different files is confusing, esp. when
> not using structs to define the file format.
>
>>> Here's what I came up with (just a sketch, commit message is lacky and the
>>> helper functions deserve a better place / name):
>>
>> I like it. Want to clean it up and submit in place of mine?
>>
>
> Will do, but it will have to wait till next week.
Thanks, both.
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2014-12-01 2:09 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-11-26 23:09 What's cooking in git.git (Nov 2014, #04; Wed, 26) Junio C Hamano
2014-11-27 5:24 ` [PATCH] pack-bitmap: do not use gcc packed attribute Jeff King
2014-11-27 18:39 ` What's cooking in git.git (Nov 2014, #04; Wed, 26) brian m. carlson
2014-11-30 8:35 ` Duy Nguyen
2014-11-30 18:18 ` brian m. carlson
2014-12-01 2:09 ` Junio C Hamano
-- strict thread matches above, loose matches on Subject: below --
2014-07-28 17:17 struct hashmap_entry packing Jeff King
2014-07-29 20:40 ` Karsten Blees
2014-08-01 22:37 ` Jeff King
2014-08-01 23:10 ` [PATCH] pack-bitmap: do not use gcc packed attribute Jeff King
2014-08-01 23:12 ` Jeff King
2014-08-04 19:19 ` Karsten Blees
2014-08-05 18:38 ` Vicent Martí
2014-08-05 18:47 ` Jeff King
2014-08-06 18:58 ` Karsten Blees
2014-08-06 19:32 ` Junio C Hamano
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).