* Re: [PATCH] http: inform about alternates-as-redirects behavior
From: Eric Wong @ 2017-03-04 6:55 UTC (permalink / raw)
To: Jeff King
Cc: Junio C Hamano, Jann Horn, Brandon Williams, git, sbeller, bburky,
jrnieder
In-Reply-To: <20170304031314.32bta4prahf7pfp7@sigill.intra.peff.net>
Jeff King <peff@peff.net> wrote:
> The warning itself:
>
> > + warning("alternate disabled by http.followRedirects!=true: %s",
>
> feels like it could use some whitespace around the "!=", but maybe
> that's just me.
Yeah, I kinda wanted to emulate the command-line syntax.
Maybe rewording it a bit and showing how to enable it will
make more sense:
warning("alternate: %s", url);
warning(" may be enabled by -c http.followRedirects=true");
As well as keeping individual lines shorter and hopefully
easier-to-read.
^ permalink raw reply
* Re: [PATCH] http: inform about alternates-as-redirects behavior
From: Jeff King @ 2017-03-04 7:41 UTC (permalink / raw)
To: Eric Wong
Cc: Junio C Hamano, Jann Horn, Brandon Williams, git, sbeller, bburky,
jrnieder
In-Reply-To: <20170304065548.GA20734@whir>
On Sat, Mar 04, 2017 at 06:55:48AM +0000, Eric Wong wrote:
> Jeff King <peff@peff.net> wrote:
> > The warning itself:
> >
> > > + warning("alternate disabled by http.followRedirects!=true: %s",
> >
> > feels like it could use some whitespace around the "!=", but maybe
> > that's just me.
>
> Yeah, I kinda wanted to emulate the command-line syntax.
>
> Maybe rewording it a bit and showing how to enable it will
> make more sense:
>
> warning("alternate: %s", url);
> warning(" may be enabled by -c http.followRedirects=true");
I kind of hoped people would look at the documentation for
followRedirects before blindly enabling it. Though I guess the
documentation doesn't really explain the possible security implications,
so maybe it doesn't matter (and they're pretty subtle anyway).
-Peff
^ permalink raw reply
* Re: [PATCH v3 0/9] Fix the early config
From: Jeff King @ 2017-03-04 7:39 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: git, Junio C Hamano, Duy Nguyen
In-Reply-To: <cover.1488562287.git.johannes.schindelin@gmx.de>
On Fri, Mar 03, 2017 at 06:31:55PM +0100, Johannes Schindelin wrote:
> Interdiff vs v2:
> [...]
> + * When we are not about to create a repository ourselves (init or
> + * clone) and when no .git/ directory was set up yet (in which case
> + * git_config_with_options() would already have picked up the
> + * repository config), we ask discover_git_directory() to figure out
> + * whether there is any repository config we should use (but unlike
> + * setup_git_directory_gently(), no global state is changed, most
> + * notably, the current working directory is still the same after
> + * the call).
> */
> - if (!startup_info->creating_repository && !have_git_dir() &&
> - discover_git_directory(&buf)) {
> + if (!have_git_dir() && discover_git_directory(&buf)) {
I think this "when we are not about to..." part of the comment is no
longer true, given the second part of the hunk.
> @@ -721,8 +721,10 @@ static const char *setup_discovered_git_dir(const char *gitdir,
> if (offset == cwd->len)
> return NULL;
>
> - /* Make "offset" point to past the '/', and add a '/' at the end */
> - offset++;
> + /* Make "offset" point past the '/' (already the case for root dirs) */
> + if (offset != offset_1st_component(cwd->buf))
> + offset++;
Nice. I was worried we would have to have a hacky "well, sometimes we
don't add one here..." code, but using offset_1st_component says
exactly what we mean.
> +/* Find GIT_DIR without changing the working directory or other global state */
> extern const char *discover_git_directory(struct strbuf *gitdir);
The parts that actually confused me were the parameters (mostly whether
gitdir was a directory to start looking in, or an output parameter). So
maybe:
/*
* Find GIT_DIR of the repository that contains the current working
* directory, without changing the working directory or other global
* state. The result is appended to gitdir. The return value is NULL
* if no repository was found, or gitdir->buf otherwise.
*/
This looks good to me aside from those few comment nits. I'm still not
sure I understand how ceil_offset works in setup_git_directory_gently_1(),
but I don't think your patch actually changed it. I can live with my
confusion.
-Peff
^ permalink raw reply
* [PATCH v2] http: inform about alternates-as-redirects behavior
From: Eric Wong @ 2017-03-04 8:36 UTC (permalink / raw)
To: Jeff King
Cc: Junio C Hamano, Jann Horn, Brandon Williams, git, sbeller, bburky,
jrnieder
In-Reply-To: <20170304074140.mzgs27jp2jer4mlv@sigill.intra.peff.net>
Jeff King <peff@peff.net> wrote:
> On Sat, Mar 04, 2017 at 06:55:48AM +0000, Eric Wong wrote:
> > Jeff King <peff@peff.net> wrote:
> > > The warning itself:
> > >
> > > > + warning("alternate disabled by http.followRedirects!=true: %s",
> > >
> > > feels like it could use some whitespace around the "!=", but maybe
> > > that's just me.
> >
> > Yeah, I kinda wanted to emulate the command-line syntax.
> >
> > Maybe rewording it a bit and showing how to enable it will
> > make more sense:
> >
> > warning("alternate: %s", url);
> > warning(" may be enabled by -c http.followRedirects=true");
>
> I kind of hoped people would look at the documentation for
> followRedirects before blindly enabling it. Though I guess the
> documentation doesn't really explain the possible security implications,
> so maybe it doesn't matter (and they're pretty subtle anyway).
You bring up a good point, perhaps just mentioning the config
key is enough to convince somebody to (v2 below).
I also think the security implications for relative alternates
on the same host would not matter, since the smart HTTP will
take them into account on the server side.
Perhaps we give http_follow_config ORable flags:
HTTP_FOLLOW_NONE = 0,
HTTP_FOLLOW_INITIAL = 0x1,
HTTP_FOLLOW_RELATIVE = 0x2,
HTTP_FOLLOW_ABSOLUTE = 0x4,
HTTP_FOLLOW_ALWAYS = 0x7,
With the default would being: HTTP_FOLLOW_INITIAL|HTTP_FOLLOW_RELATIVE
(but I suppose that's a patch for another time)
----------8<-----------
From: Eric Wong <e@80x24.org>
Subject: [PATCH] http: inform about alternates-as-redirects behavior
It is disconcerting for users to not notice the behavior
change in handling alternates from commit cb4d2d35c4622ec2
("http: treat http-alternates like redirects")
Give the user a hint about the config option so they can
see the URL and decide whether or not they want to enable
http.followRedirects in their config.
Signed-off-by: Eric Wong <e@80x24.org>
---
http-walker.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/http-walker.c b/http-walker.c
index b34b6ace7..6396cebe5 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -168,6 +168,11 @@ static int is_alternate_allowed(const char *url)
};
int i;
+ if (http_follow_config != HTTP_FOLLOW_ALWAYS) {
+ warning("alternate disabled by http.followRedirects: %s", url);
+ return 0;
+ }
+
for (i = 0; i < ARRAY_SIZE(protocols); i++) {
const char *end;
if (skip_prefix(url, protocols[i], &end) &&
@@ -331,9 +336,6 @@ static void fetch_alternates(struct walker *walker, const char *base)
struct alternates_request alt_req;
struct walker_data *cdata = walker->data;
- if (http_follow_config != HTTP_FOLLOW_ALWAYS)
- return;
-
/*
* If another request has already started fetching alternates,
* wait for them to arrive and return to processing this request's
--
EW
^ permalink raw reply related
* Re: [PATCH v2] http: inform about alternates-as-redirects behavior
From: Jeff King @ 2017-03-04 8:45 UTC (permalink / raw)
To: Eric Wong
Cc: Junio C Hamano, Jann Horn, Brandon Williams, git, sbeller, bburky,
jrnieder
In-Reply-To: <20170304083645.GA24694@whir>
On Sat, Mar 04, 2017 at 08:36:45AM +0000, Eric Wong wrote:
> I also think the security implications for relative alternates
> on the same host would not matter, since the smart HTTP will
> take them into account on the server side.
It depends on the host whether all of the repos on it have the same
security domain or not. A site like github.com hosts both public and
private repositories, and you do not want a public repo redirecting to
the private one to get objects.
Of course, that depends on untrusted users being able to configure
server-side alternates, which GitHub certainly would not let you do. I
would hope other multi-user hosting sites behave similarly (most hosting
sites do not seem to allow dumb http at all).
> Perhaps we give http_follow_config ORable flags:
>
> HTTP_FOLLOW_NONE = 0,
> HTTP_FOLLOW_INITIAL = 0x1,
> HTTP_FOLLOW_RELATIVE = 0x2,
> HTTP_FOLLOW_ABSOLUTE = 0x4,
> HTTP_FOLLOW_ALWAYS = 0x7,
>
> With the default would being: HTTP_FOLLOW_INITIAL|HTTP_FOLLOW_RELATIVE
> (but I suppose that's a patch for another time)
I don't have a real problem with breaking it down that way, if somebody
wants to make a patch. Mostly the reason I didn't do so is that I don't
think http-alternates are in common use these days, since smart-http is
much more powerful.
> ----------8<-----------
> From: Eric Wong <e@80x24.org>
> Subject: [PATCH] http: inform about alternates-as-redirects behavior
This v2 looks fine to me.
-Peff
^ permalink raw reply
* Re: Delta compression not so effective
From: Marius Storm-Olsen @ 2017-03-04 8:27 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git Mailing List
In-Reply-To: <CA+55aFxxQUixAJWXkUgVvDNCHD4LuYYuQRTE7dJ_OZTo9Gxqew@mail.gmail.com>
On 3/1/2017 18:43, Linus Torvalds wrote:
>> So, this repo must be knocking several parts of Git's insides. I was curious
>> about why it was so slow on the writing objects part, since the whole repo
>> is on a 4x RAID 5, 7k spindels. Now, they are not SSDs sure, but the thing
>> has ~400MB/s continuous throughput available.
>>
>> iostat -m 5 showed trickle read/write to the process, and 80-100% CPU single
>> thread (since the "write objects" stage is single threaded, obviously).
>
> So the writing phase isn't multi-threaded because it's not expected to
> matter. But if you can't even generate deltas, you aren't just
> *writing* much more data, you're compressing all that data with zlib
> too.
>
> So even with a fast disk subsystem, you won't even be able to saturate
> the disk, simply because the compression will be slower (and
> single-threaded).
I did a simple
$ time zip -r repo.zip repo/
...
total bytes=219353596620, compressed=214310715074 -> 2% savings
real 154m6.323s
user 133m5.209s
sys 5m5.338s
also using a single thread + same disk, as git repack. But if you
compare it to the numbers below, it's 2.6hrs with zip vs 14.2hrs
(1:5.5). So it can't just be the overhead of having to compress the full
blobs, due to lacking delta..
>> Filenames are fairly static, and the bulk of the 6000 biggest non-delta'ed
>> blobs are the same DLLs (multiple of them)
>
> I think the first thing you should test is to repack with fewer
> threads, and a bigger pack window. Do somethinig like
>
> -c pack.threads=4 --window-memory=30g
>
> instead. Just to see if that starts finding deltas.
I reran the repack with the options above (dropping the zlib=9, as you
suggested)
$ time git -c pack.threads=4 repack -a -d -F \
--window=350 --depth=250 --window-memory=30g
Delta compression using up to 4 threads.
Compressing objects: 100% (609413/609413)
Writing objects: 100% (666515/666515), done.
Total 666515 (delta 499585), reused 0 (delta 0)
real 850m3.473s
user 897m36.280s
sys 10m8.824s
and ended up with
$ du -sh .
205G .
In other words, going from 6G to 30G window didn't help a lick on
finding deltas for those binaries. (205G was what I had with the
non-aggressive 'git gc', before zlib=9 repack.)
BUT, oddly enough, even if the new size if almost identical to the
previous version without zlib=9,
git verify-pack --verbose
objects/pack/pack-29b06ae4d458ac03efd98b330702d30e851b2933.idx | sort
-k3n | tail -n15
gives me a VERY different list than before
17e5b2146311256dc8317d6e0ed1291363c31a76 blob 673399562 110248747
190398904084
04c881d9069eab3bd0d50dd48a047a60f79cc415 blob 673863358 111710559
188818868865
fdcabd75aeda86ce234d6e43b54d27d993acddcd blob 674523614 111956017
185706433825
d8815033d1b00b151ae762be8a69ffa35f55c4b4 blob 675286758 112099638
185153570292
997e0b9d3bcf440af10c7bbe535a597ca46c492c blob 678274978 112654668
184041692883
dfed141679e5c33caaa921cbe1595a24967a3c2c blob 681692132 113121410
186753502634
76a4000e71cd5b85f2265e02eb876acf1f33cc55 blob 682673430 112743915
184563542298
81e7292c4d2da2d2d236fbfaa572b6c4e8d787f4 blob 684543130 112797325
181805773038
991184c60e1fc6b2721bf40f181012b72b10d02d blob 684543130 112796892
182344388066
0e9269f4abd1440addd05d4f964c96d74d11cd89 blob 684547270 112809074
181070719237
6019b6d09759cf5adeac678c8b56d177803a0486 blob 684547270 112809336
180517242193
70a5f70bd205329472d6f9c660eb3f7d207a596e blob 686852038 112873611
183520467528
e86a0064d9652be9f5e3a877b11a665f64198ecd blob 686852038 112874133
182893219377
bae8de0555be5b1ffa0988cbc6cba698f6745c26 blob 894041802 137223252
2355250324
94dc773600e03ac1e6f3ab077b70b8297325ad77 blob 945197364 145219485
16560137220
compared to the last 3 entries of the previous pack
e9916da851962265a9d5b099e72f60659a74c144 blob 170113524 73514361
966299538
f7bf1313752deb1bae592cc7fc54289aea87ff19 blob 170113524 70756581
1039814687
8afc6f2a51f0fa1cc4b03b8d10c70599866804ad blob 248959314 237612609
606692699
> So the first thing you might want to do is to just print out the
> objects after sorting them, and before it starts trying to finsd
> deltas.
...
> and notice that QSORT() line: that's what sorts the objects. You can
> do something like
>
> for (i = 0; i < n; i++)
> show_object_entry_details(delta_list[i]);
I did
fprintf(stderr, "%s %u %lu\n",
sha1_to_hex(delta_list[i]->idx.sha1),
delta_list[i]->hash,
delta_list[i]->size);
I assume that's correct?
> In fact, if your data is not *so* sensitive, and you're ok with making
> the one-line commit logs and the filenames public, you could make just
> those things available, and maybe I'll have time to look at it.
I've removed all commit messages, and "sanitized" some filepaths etc, so
name hashes won't match what's reported, but that should be fine. (the
object_entry->hash seems to be just a trivial uint32 hash for sorting
anyways)
I really don't want the files on the mailinglist, so I'll send you a
link directly. However, small snippets for public discussions about
potential issues would be fine, obviously.
BUT, if I look at the last 3 entries of the sorted git verify-pack
output, and look for them in the 'git log --oneline --raw -R
--abbrev=40' output, I get:
:100644 100644 991184c60e1fc6b2721bf40f181012b72b10d02d
e86a0064d9652be9f5e3a877b11a665f64198ecd M
extern/win/FlammableV3/x64/lib/FlameProxyLibD.lib
:100644 000000 bae8de0555be5b1ffa0988cbc6cba698f6745c26
0000000000000000000000000000000000000000 D
extern/win/gdal-2.0.0/lib/x64/Debug/libgdal.lib
:000000 100644 0000000000000000000000000000000000000000
94dc773600e03ac1e6f3ab077b70b8297325ad77 A
extern/win/gdal-2.0.0/lib/x64/Debug/gdal.lib
while I cannot find ANY of them in the delta_list output?? Shouldn't
delta_list contain all objects, sorted by some heuristics? Or is the
delta_list already here limited by some other metric, before the QSORT?
Also note that the 'git log --oneline --raw -R --abbrev=40' only gave me
the log for trunk, so for the second last object, must have been added
in a branch, and deleted on trunk; so I could only see the deletion of
that object in the output.
You might get an idea for how to easily create a repo which reproduces
the issue, and which would highlight it more easily for the ML.
I was thinking of maybe scripting up
make install prefix=extern
for each Git release, and rewrite trunk history with extern/ binary
commits at the time of each tag; maybe that would show the same
behavior? But then again, most of the binaries are just copies of each
other, and only ~10M, so probably not a big win.
Thanks!
--
.marius
^ permalink raw reply
* GSoC 2017
From: Valery Tolstov @ 2017-03-04 11:00 UTC (permalink / raw)
To: git
Hi.
Want to participate in GSoC 2017 with Git.
Can I choose "Make 'git tag --contains <id>' less chatty" as my microproject?
Thanks.
^ permalink raw reply
* Re: GSoC 2017
From: Thomas Gummerer @ 2017-03-04 11:45 UTC (permalink / raw)
To: Valery Tolstov; +Cc: git
In-Reply-To: <5334661488625223@web28h.yandex.ru>
Hi,
On 03/04, Valery Tolstov wrote:
> Hi.
>
> Want to participate in GSoC 2017 with Git.
> Can I choose "Make 'git tag --contains <id>' less chatty" as my microproject?
Feel free to choose any microproject from the list. A good way to
start is to check whether other students already attempted the same
microproject before starting with it so you don't duplicate any
work. You can check for that in the mailing list archives, available
at [1].
[1]: https://public-inbox.org/git/
> Thanks.
Good luck!
^ permalink raw reply
* git init --separate-git-dir does not update symbolic .git links for submodules
From: Sven Strickroth @ 2017-03-04 12:15 UTC (permalink / raw)
To: Git List, Junio C Hamano
Hi,
clone or have a repository with submodules.
Now issue "git init --separate-git-dir ../test" which moves the content
of the .git direcotry to the ../test directory and replaces the .git
directory with a .git file "link".
However, for all submodules the .git-file is not updated and still
points to the .git directory of the parent repository which now is a file.
This issue is present for me in Git 2.12.0.
--
Best regards,
Sven Strickroth
PGP key id F5A9D4C4 @ any key-server
^ permalink raw reply
* Re: git init --separate-git-dir does not update symbolic .git links for submodules
From: Valery Tolstov @ 2017-03-04 14:15 UTC (permalink / raw)
To: sven; +Cc: git, gitster
Looking for microproject ideas for GSoC.
Would this issue be suitable as the microproject?
Thanks,
Valery Tolstov
^ permalink raw reply
* Re: What's cooking in git.git (Mar 2017, #02; Fri, 3)
From: Stephan Beyer @ 2017-03-04 14:35 UTC (permalink / raw)
To: Pranit Bauva; +Cc: Junio C Hamano, git
In-Reply-To: <xmqqh93a6iy6.fsf@gitster.mtv.corp.google.com>
Hi Pranit,
On 03/04/2017 12:26 AM, Junio C Hamano wrote:
> --------------------------------------------------
> [Stalled]
[...]
>
> * pb/bisect (2017-02-18) 28 commits
> - fixup! bisect--helper: `bisect_next_check` & bisect_voc shell function in C
> - bisect--helper: remove the dequote in bisect_start()
> - bisect--helper: retire `--bisect-auto-next` subcommand
> - bisect--helper: retire `--bisect-autostart` subcommand
> - bisect--helper: retire `--bisect-write` subcommand
> - bisect--helper: `bisect_replay` shell function in C
> - bisect--helper: `bisect_log` shell function in C
> - bisect--helper: retire `--write-terms` subcommand
> - bisect--helper: retire `--check-expected-revs` subcommand
> - bisect--helper: `bisect_state` & `bisect_head` shell function in C
> - bisect--helper: `bisect_autostart` shell function in C
> - bisect--helper: retire `--next-all` subcommand
> - bisect--helper: retire `--bisect-clean-state` subcommand
> - bisect--helper: `bisect_next` and `bisect_auto_next` shell function in C
> - t6030: no cleanup with bad merge base
> - bisect--helper: `bisect_start` shell function partially in C
> - bisect--helper: `get_terms` & `bisect_terms` shell function in C
> - bisect--helper: `bisect_next_check` & bisect_voc shell function in C
> - bisect--helper: `check_and_set_terms` shell function in C
> - bisect--helper: `bisect_write` shell function in C
> - bisect--helper: `is_expected_rev` & `check_expected_revs` shell function in C
> - bisect--helper: `bisect_reset` shell function in C
> - wrapper: move is_empty_file() and rename it as is_empty_or_missing_file()
> - t6030: explicitly test for bisection cleanup
> - bisect--helper: `bisect_clean_state` shell function in C
> - bisect--helper: `write_terms` shell function in C
> - bisect: rewrite `check_term_format` shell function in C
> - bisect--helper: use OPT_CMDMODE instead of OPT_BOOL
>
> Move more parts of "git bisect" to C.
>
> Expecting a reroll.
I guess you are short on time but I am hoping that you are still going
to send a reroll to the list (probably on top of [1]?). This is because
I would like to start to work on rerolling my bisect patches from last
year [2] ... but to avoid a mess of merge conflicts, I am waiting until
pb/bisect found its way into "next". (There were also recent discussions
on other bisect strategies [3] and it's probably only a matter of time
until a new big patchset on bisect--helper comes up...)
Cheers
Stephan
[1]:
https://public-inbox.org/git/xmqqvarq9vzo.fsf@gitster.mtv.corp.google.com/
[2]:
https://public-inbox.org/git/1460294354-7031-1-git-send-email-s-beyer@gmx.net/
[3]:
https://public-inbox.org/git/CABEd3j8m5D=bBbUD+uzvE9c8AwdBEM79Np7Pnin-RLL=Hjq06A@mail.gmail.com/
^ permalink raw reply
* Re: [PATCH] http: inform about alternates-as-redirects behavior
From: Ramsay Jones @ 2017-03-04 15:06 UTC (permalink / raw)
To: Jeff King, Eric Wong
Cc: Junio C Hamano, Jann Horn, Brandon Williams, git, sbeller, bburky,
jrnieder
In-Reply-To: <20170304031314.32bta4prahf7pfp7@sigill.intra.peff.net>
On 04/03/17 03:13, Jeff King wrote:
> On Sat, Mar 04, 2017 at 01:35:04AM +0000, Eric Wong wrote:
[snip]
>
> The warning itself:
>
>> + warning("alternate disabled by http.followRedirects!=true: %s",
>
> feels like it could use some whitespace around the "!=", but maybe
> that's just me.
It's probably just me (too), but I think it would read better
without having '!=true' in the message at all. ;-)
ATB,
Ramsay Jones
^ permalink raw reply
* Re: [PATCH v1] Travis: also test on 32-bit Linux
From: Lars Schneider @ 2017-03-04 17:23 UTC (permalink / raw)
To: Junio C Hamano, allan.x.xavier, vegard.nossum, Jeff King
Cc: Johannes Schindelin, Git Mailing List
In-Reply-To: <CAPc5daW=gtN18JZTQMqUje5fxL4oNdTucB0dXFbybPRJggPBUw@mail.gmail.com>
> On 04 Mar 2017, at 05:11, Junio C Hamano <gitster@pobox.com> wrote:
>
> On Fri, Mar 3, 2017 at 4:03 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>
>> I only recently started looking at Travis logs, so I cannot tell if
>> it is just a usual flake (e.g. some builds a few days ago seems to
>> have failed due to not being able to check out the tree being
>> tested, which I do not think is our fault) that we shouldn't worry
>> about, or if it is a sign of a real problem.
>
> Tonight's pushout also seems to stall the same way. Dscho's
> unversioned one didn't exhibit the problem?
> https://travis-ci.org/git/git/jobs/206811396
Did Travis find our first 32bit bug? I briefly looked into it
and the following new topic on pu seems to cause the issue:
http://public-inbox.org/git/20170302172902.16850-1-allan.x.xavier@oracle.com/
https://github.com/git/git/commit/aaae0bf787f09ba102f69c3cf85d37e6554ab9fd
The "git log" call in the new 4211 test fails with:
*** Error in `/usr/src/git/git': malloc: top chunk is corrupt: 0x09ff4a78 ***
More output here:
https://travis-ci.org/larsxschneider/git/builds/207715343
>> Unrelated to linux-32, the same build has hard failure with Apple
>> clang in t0021 with the rot13-filter.pl thing, by the way.
>
> This one may be a Heisenbug which may indicate some raciness in the
> clean/smudge filter protocol.
> The latest build of 'pu' https://travis-ci.org/git/git/jobs/207550171 seems to
> have passed OK.
I agree, there seems to be some kind of race condition in
"t0021.15 - required process filter should filter data". I try to
look into it.
- Lars
^ permalink raw reply
* Re: GSoC 2017
From: Thomas Gummerer @ 2017-03-04 17:26 UTC (permalink / raw)
To: Valery Tolstov; +Cc: git
In-Reply-To: <20170304114504.GA27158@hank>
On 03/04, Thomas Gummerer wrote:
> Hi,
>
> On 03/04, Valery Tolstov wrote:
> > Hi.
> >
> > Want to participate in GSoC 2017 with Git.
> > Can I choose "Make 'git tag --contains <id>' less chatty" as my microproject?
>
> Feel free to choose any microproject from the list. A good way to
> start is to check whether other students already attempted the same
> microproject before starting with it so you don't duplicate any
> work. You can check for that in the mailing list archives, available
> at [1].
And I meant to say, if nobody started working on the micro-project yet
(and I don't recall anything doing so, but better double check), feel
free to just pick it up.
If somebody was already working on it might be better to try and find
a micro-project that nobody is working on yet, as that is going to
be a better learning experience for you, and allows us to get to know
you better.
> [1]: https://public-inbox.org/git/
>
> > Thanks.
>
> Good luck!
^ permalink raw reply
* Re: What's cooking in git.git (Mar 2017, #02; Fri, 3)
From: Lars Schneider @ 2017-03-04 17:32 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <xmqqh93a6iy6.fsf@gitster.mtv.corp.google.com>
> On 04 Mar 2017, at 00:26, Junio C Hamano <gitster@pobox.com> wrote:
>
>
> * ls/filter-process-delayed (2017-01-08) 1 commit
> . convert: add "status=delayed" to filter process protocol
>
> Ejected, as does not build when merged to 'pu'.
I send v2 [1] where I tried to address the points in your feedback [2].
v2 not the final roll. My goal for v2 is to get the interface
to convert.h right. Could you take a quick look at the changes
in the following files and tell me if this is the right direction to go?
entry.c
builtin/checkout.c
unpack-trees.c
Thanks,
Lars
[1] http://public-inbox.org/git/20170226184816.30010-1-larsxschneider@gmail.com/
[2] http://public-inbox.org/git/xmqqa8b115ll.fsf@gitster.mtv.corp.google.com/
^ permalink raw reply
* Re: [PATCH v1] Travis: also test on 32-bit Linux
From: Vegard Nossum @ 2017-03-04 18:08 UTC (permalink / raw)
To: Lars Schneider, Junio C Hamano, allan.x.xavier, Jeff King
Cc: Johannes Schindelin, Git Mailing List
In-Reply-To: <2205F1A7-A694-4F40-B994-D68C3947F2BB@gmail.com>
On 04/03/2017 18:23, Lars Schneider wrote:
> Did Travis find our first 32bit bug? I briefly looked into it
> and the following new topic on pu seems to cause the issue:
>
> http://public-inbox.org/git/20170302172902.16850-1-allan.x.xavier@oracle.com/
> https://github.com/git/git/commit/aaae0bf787f09ba102f69c3cf85d37e6554ab9fd
>
> The "git log" call in the new 4211 test fails with:
> *** Error in `/usr/src/git/git': malloc: top chunk is corrupt: 0x09ff4a78 ***
>
> More output here:
> https://travis-ci.org/larsxschneider/git/builds/207715343
It looks like it's hitting the bug the patch is supposed to fix.
Are you quite sure it's running the "git" binary that was just built (as
opposed to e.g. the system binary installed inside the container)?
Vegard
^ permalink raw reply
* [RFC 0/4] Shallow clones with on-demand fetch
From: Mark Thomas @ 2017-03-04 19:18 UTC (permalink / raw)
To: git; +Cc: Mark Thomas
Hello everyone,
This is an RFC for an enhancement to shallow repositories to make them
behave more like full clones.
I was inspired a bit by Microsoft's announcement of their Git VFS. I
saw that people have talked in the past about making git fetch objects
from remotes as they are needed, and decided to give it a try.
The patch series adds a "--on-demand" option to git clone, which, when
used in conjunction with the existing shallow clone operations, clones
the full history of the repository's commits, but only the files that
would be included in the shallow clone.
When a file that is missing is required, git requests the file on-demand
from the remote, via a new 'upload-file' service.
Public git servers are unlikely to want to enable this, due to the
addition load it may cause, but within an organization's own network, it
will allow full access to the repository history without needing a full
initial clone.
The patch set is in four parts:
1:
Adds the "upload-file" command, which starts a new protocol
conversation with the client allowing it to request file info and
file contents. The connection is kept open so that the client
can make as many requests as it likes. The client terminates the
connection by sending a packet containing "end".
2:
Adds the ability for file info and content to be requested from
the remote if the file cannot be found in any pack, or loose in
the repository. Currently this only looks at the default remote,
but the intention is this would be configurable.
3:
Adds the "on-demand" capability to "upload-pack". When a client
requests this capability, "upload-pack" includes in the pack
all commits, even those that would normally be dropped by the
shallow clone.
4:
Adds the "--on-demand" option to clone, to request a shallow
clone.
This is a proof-of-concept, so it is in no way complete. It contains a
few hacks to make it work, but these can be ironed out with a bit more
work. What I have so far is sufficient to try out the idea. I'd like
to get people's opinions on it before I spend any more time working on
it, plus also I'm not very familiar with the git codebase, so some help
would be appreciated.
As an example, the Linux repository currently stands at 2.0GB of packed
data. A "git clone --shallow-since=2016-01-01 --on-demand" is only
561MB, and yet remains fully functional. A git blame on the Makefile,
for example, shows all changes to the file, right back to Linus's
original commit in 2005.
Still to do:
- Fix up the hacks and make everything work correctly.
- Make fetching of further updates work correctly.
- Store the retrieved files in an LRU cache, possibly with the option
of storing them in the main repo data, too.
- Add a gc/enshallow operation to make the repo shallower by forgetting
old files, or moving them to the LRU cache.
- Add configurable remote to fetch from.
- Documentation.
- Much more.
Please let me know what you think, and if an experienced git developer
would like to help out with finishing this, that would be even better.
Mark Thomas (4):
upload-file: Add upload-file command
on-demand: Fetch missing files from remote
upload-pack: Send all commits if client requests on-demand
clone: Request on-demand shallow clones
.gitignore | 1 +
Makefile | 3 +
builtin/clone.c | 7 +-
builtin/pack-objects.c | 26 ++++++-
cache-tree.c | 2 +-
cache.h | 3 +-
daemon.c | 6 ++
fetch-pack.c | 3 +
fetch-pack.h | 1 +
list-objects.c | 12 ++--
list-objects.h | 13 +++-
object.h | 1 +
on_demand.c | 183 +++++++++++++++++++++++++++++++++++++++++++++++++
on_demand.h | 12 ++++
sha1_file.c | 8 ++-
shallow.c | 2 +-
transport.c | 3 +
transport.h | 4 ++
upload-file.c | 87 +++++++++++++++++++++++
upload-pack.c | 8 ++-
20 files changed, 370 insertions(+), 15 deletions(-)
create mode 100644 on_demand.c
create mode 100644 on_demand.h
create mode 100644 upload-file.c
--
2.7.4
^ permalink raw reply
* [RFC 3/4] upload-pack: Send all commits if client requests on-demand
From: Mark Thomas @ 2017-03-04 19:19 UTC (permalink / raw)
To: git; +Cc: Mark Thomas
In-Reply-To: <20170304191901.9622-1-markbt@efaref.net>
Signed-off-by: Mark Thomas <markbt@efaref.net>
---
builtin/pack-objects.c | 26 ++++++++++++++++++++++++--
list-objects.c | 12 +++++++-----
list-objects.h | 13 ++++++++++++-
object.h | 1 +
on_demand.c | 26 ++++++++++++++++++++++++++
on_demand.h | 4 ++++
upload-pack.c | 8 +++++++-
7 files changed, 81 insertions(+), 9 deletions(-)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index f294dcf..c8b2503 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -24,6 +24,7 @@
#include "sha1-array.h"
#include "argv-array.h"
#include "mru.h"
+#include "on_demand.h"
static const char *pack_usage[] = {
N_("git pack-objects --stdout [<options>...] [< <ref-list> | < <object-list>]"),
@@ -77,6 +78,8 @@ static unsigned long cache_max_small_delta_size = 1000;
static unsigned long window_memory_limit = 0;
+static int send_all_commits;
+
/*
* stats
*/
@@ -2750,12 +2753,15 @@ static void record_recent_commit(struct commit *commit, void *data)
static void get_object_list(int ac, const char **av)
{
struct rev_info revs;
+ struct rev_info revs2;
char line[1000];
int flags = 0;
init_revisions(&revs, NULL);
+ init_revisions(&revs2, NULL);
save_commit_buffer = 0;
setup_revisions(ac, av, &revs, NULL);
+ setup_revisions(ac, av, &revs2, NULL);
/* make sure shallows are read */
is_repository_shallow();
@@ -2776,7 +2782,10 @@ static void get_object_list(int ac, const char **av)
unsigned char sha1[20];
if (get_sha1_hex(line + 10, sha1))
die("not an SHA-1 '%s'", line + 10);
- register_shallow(sha1);
+ if (send_all_commits)
+ register_on_demand_cutoff(sha1);
+ else
+ register_shallow(sha1);
use_bitmap_index = 0;
continue;
}
@@ -2784,6 +2793,8 @@ static void get_object_list(int ac, const char **av)
}
if (handle_revision_arg(line, &revs, flags, REVARG_CANNOT_BE_FILENAME))
die("bad revision '%s'", line);
+ if (handle_revision_arg(line, &revs2, flags, REVARG_CANNOT_BE_FILENAME))
+ die("bad revision '%s'", line);
}
if (use_bitmap_index && !get_object_list_from_bitmap(&revs))
@@ -2792,7 +2803,16 @@ static void get_object_list(int ac, const char **av)
if (prepare_revision_walk(&revs))
die("revision walk setup failed");
mark_edges_uninteresting(&revs, show_edge);
- traverse_commit_list(&revs, show_commit, show_object, NULL);
+
+ if (send_all_commits) {
+ revs2.include_check = on_demand_include_check;
+ traverse_commit_list(&revs2, on_demand_show_commit_tree, NULL,
+ NULL);
+ reset_revision_walk();
+ }
+
+ traverse_commit_list_extended(&revs, show_commit, show_object,
+ on_demand_show_tree_check, NULL);
if (unpack_unreachable_expiration) {
revs.ignore_missing_links = 1;
@@ -2928,6 +2948,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
N_("use a bitmap index if available to speed up counting objects")),
OPT_BOOL(0, "write-bitmap-index", &write_bitmap_index,
N_("write a bitmap index together with the pack index")),
+ OPT_BOOL(0, "send-all-commits", &send_all_commits,
+ N_("send all commits for on-demand shallow fetches")),
OPT_END(),
};
diff --git a/list-objects.c b/list-objects.c
index f3ca6aa..2607549 100644
--- a/list-objects.c
+++ b/list-objects.c
@@ -183,10 +183,11 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)
add_pending_object(revs, &tree->object, "");
}
-void traverse_commit_list(struct rev_info *revs,
- show_commit_fn show_commit,
- show_object_fn show_object,
- void *data)
+void traverse_commit_list_extended(struct rev_info *revs,
+ show_commit_fn show_commit,
+ show_object_fn show_object,
+ show_tree_check_fn show_tree_check,
+ void *data)
{
int i;
struct commit *commit;
@@ -198,7 +199,8 @@ void traverse_commit_list(struct rev_info *revs,
* an uninteresting boundary commit may not have its tree
* parsed yet, but we are not going to show them anyway
*/
- if (commit->tree)
+ if (show_object && commit->tree &&
+ (!show_tree_check || show_tree_check(commit, data)))
add_pending_tree(revs, commit->tree);
show_commit(commit, data);
}
diff --git a/list-objects.h b/list-objects.h
index 0cebf85..e80dc8c 100644
--- a/list-objects.h
+++ b/list-objects.h
@@ -3,7 +3,18 @@
typedef void (*show_commit_fn)(struct commit *, void *);
typedef void (*show_object_fn)(struct object *, const char *, void *);
-void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);
+typedef int (*show_tree_check_fn)(struct commit *, void *);
+void traverse_commit_list_extended(struct rev_info *, show_commit_fn,
+ show_object_fn, show_tree_check_fn, void *);
+
+inline void traverse_commit_list(struct rev_info *revs,
+ show_commit_fn show_commit,
+ show_object_fn show_object,
+ void *data)
+{
+ traverse_commit_list_extended(revs, show_commit, show_object, NULL,
+ data);
+}
typedef void (*show_edge_fn)(struct commit *);
void mark_edges_uninteresting(struct rev_info *, show_edge_fn);
diff --git a/object.h b/object.h
index f52957d..25177fd 100644
--- a/object.h
+++ b/object.h
@@ -38,6 +38,7 @@ struct object_array {
* http-push.c: 16-----19
* commit.c: 16-----19
* sha1_name.c: 20
+ * on_demand.c: 21-22
*/
#define FLAG_BITS 27
diff --git a/on_demand.c b/on_demand.c
index a0aaf18..c72d7a5 100644
--- a/on_demand.c
+++ b/on_demand.c
@@ -155,3 +155,29 @@ int object_info_on_demand(const unsigned char *sha1, struct object_info *oi)
die("git on-demand: protocol error, "
"unexpected response: '%s'", line);
}
+
+#define ON_DEMAND_CUTOFF (1u << 21)
+#define ON_DEMAND_SHOW_TREE (1u << 22)
+
+void register_on_demand_cutoff(const unsigned char *sha1)
+{
+ struct commit *commit = lookup_commit(sha1);
+ if (commit)
+ commit->object.flags |= ON_DEMAND_CUTOFF;
+}
+
+int on_demand_include_check(struct commit *commit, void *data)
+{
+ return !(commit->object.flags & ON_DEMAND_CUTOFF);
+}
+
+void on_demand_show_commit_tree(struct commit *commit, void *data)
+{
+ commit->object.flags |= ON_DEMAND_SHOW_TREE;
+}
+
+int on_demand_show_tree_check(struct commit *commit, void *data)
+{
+ return !!(commit->object.flags &
+ (ON_DEMAND_SHOW_TREE|ON_DEMAND_CUTOFF));
+}
diff --git a/on_demand.h b/on_demand.h
index 09a8072..7bbb523 100644
--- a/on_demand.h
+++ b/on_demand.h
@@ -4,5 +4,9 @@
void *read_remote_on_demand(const unsigned char *sha1, enum object_type *type,
unsigned long *size);
int object_info_on_demand(const unsigned char *sha1, struct object_info *oi);
+void register_on_demand_cutoff(const unsigned char *sha1);
+int on_demand_include_check(struct commit *commit, void *data);
+void on_demand_show_commit_tree(struct commit *commit, void *data);
+int on_demand_show_tree_check(struct commit *commit, void *data);
#endif
diff --git a/upload-pack.c b/upload-pack.c
index 7597ba3..1b552b4 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -42,6 +42,7 @@ static int multi_ack;
static int no_done;
static int use_thin_pack, use_ofs_delta, use_include_tag;
static int no_progress, daemon_mode;
+static int send_all_commits;
/* Allow specifying sha1 if it is a ref tip. */
#define ALLOW_TIP_SHA1 01
/* Allow request of a sha1 if it is reachable from a ref (possibly hidden ref). */
@@ -130,6 +131,8 @@ static void create_pack_file(void)
argv_array_push(&pack_objects.args, "--delta-base-offset");
if (use_include_tag)
argv_array_push(&pack_objects.args, "--include-tag");
+ if (send_all_commits)
+ argv_array_push(&pack_objects.args, "--send-all-commits");
pack_objects.in = -1;
pack_objects.out = -1;
@@ -820,6 +823,8 @@ static void receive_needs(void)
no_progress = 1;
if (parse_feature_request(features, "include-tag"))
use_include_tag = 1;
+ if (parse_feature_request(features, "on-demand"))
+ send_all_commits = 1;
o = parse_object(sha1_buf);
if (!o)
@@ -924,7 +929,8 @@ static int send_ref(const char *refname, const struct object_id *oid,
{
static const char *capabilities = "multi_ack thin-pack side-band"
" side-band-64k ofs-delta shallow deepen-since deepen-not"
- " deepen-relative no-progress include-tag multi_ack_detailed";
+ " deepen-relative no-progress include-tag multi_ack_detailed"
+ " on-demand";
const char *refname_nons = strip_namespace(refname);
struct object_id peeled;
--
2.7.4
^ permalink raw reply related
* [RFC 4/4] clone: Request on-demand shallow clones
From: Mark Thomas @ 2017-03-04 19:19 UTC (permalink / raw)
To: git; +Cc: Mark Thomas
In-Reply-To: <20170304191901.9622-1-markbt@efaref.net>
Add the --on-demand option to git-clone, which, when used in combination
with the existing shallow clone options, requests an on-demand shallow
clone.
An on-demand shallow clone contains all commits from all history, but
the commits that would normally be omitted in the shallow clone do not
have their trees or blobs in the repository. Instead, they will be
fetched on-demand from the remote.
Signed-off-by: Mark Thomas <markbt@efaref.net>
---
builtin/clone.c | 7 ++++++-
cache-tree.c | 2 +-
fetch-pack.c | 3 +++
fetch-pack.h | 1 +
shallow.c | 2 +-
transport.c | 3 +++
transport.h | 4 ++++
7 files changed, 19 insertions(+), 3 deletions(-)
diff --git a/builtin/clone.c b/builtin/clone.c
index 3f63edb..7541016 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -40,6 +40,7 @@ static const char * const builtin_clone_usage[] = {
static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
+static int option_on_demand;
static int option_shallow_submodules;
static int deepen;
static char *option_template, *option_depth, *option_since;
@@ -100,6 +101,8 @@ static struct option builtin_clone_options[] = {
N_("create a shallow clone since a specific time")),
OPT_STRING_LIST(0, "shallow-exclude", &option_not, N_("revision"),
N_("deepen history of shallow clone, excluding rev")),
+ OPT_BOOL(0, "on-demand", &option_on_demand,
+ N_("Make shallow clone an on-demand clone")),
OPT_BOOL(0, "single-branch", &option_single_branch,
N_("clone only one branch, HEAD or --branch")),
OPT_BOOL(0, "shallow-submodules", &option_shallow_submodules,
@@ -1045,6 +1048,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
if (option_not.nr)
transport_set_option(transport, TRANS_OPT_DEEPEN_NOT,
(const char *)&option_not);
+ if (option_on_demand)
+ transport_set_option(transport, TRANS_OPT_ON_DEMAND, "1");
if (option_single_branch)
transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, "1");
@@ -1118,7 +1123,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
transport_fetch_refs(transport, mapped_refs);
update_remote_refs(refs, mapped_refs, remote_head_points_at,
- branch_top.buf, reflog_msg.buf, transport, !is_local);
+ branch_top.buf, reflog_msg.buf, transport, !is_local && !option_on_demand);
update_head(our_head_points_at, remote_head, reflog_msg.buf);
diff --git a/cache-tree.c b/cache-tree.c
index 345ea35..10b14fe 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -356,7 +356,7 @@ static int update_one(struct cache_tree *it,
}
if (mode != S_IFGITLINK && !missing_ok && !has_sha1_file(sha1)) {
strbuf_release(&buffer);
- if (expected_missing)
+ if (expected_missing || 1 /*** FIXME: markbt temp hack, to allow missing files ***/)
return -1;
return error("invalid object %06o %s for '%.*s'",
mode, sha1_to_hex(sha1), entlen+baselen, path);
diff --git a/fetch-pack.c b/fetch-pack.c
index e0f5d5c..1dd4823 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -372,6 +372,7 @@ static int find_common(struct fetch_pack_args *args,
if (prefer_ofs_delta) strbuf_addstr(&c, " ofs-delta");
if (deepen_since_ok) strbuf_addstr(&c, " deepen-since");
if (deepen_not_ok) strbuf_addstr(&c, " deepen-not");
+ if (args->on_demand) strbuf_addf(&c, " on-demand");
if (agent_supported) strbuf_addf(&c, " agent=%s",
git_user_agent_sanitized());
packet_buf_write(&req_buf, "want %s%s\n", remote_hex, c.buf);
@@ -936,6 +937,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
die(_("Server does not support --shallow-exclude"));
if (!server_supports("deepen-relative") && args->deepen_relative)
die(_("Server does not support --deepen"));
+ if (!server_supports("on-demand") && args->on_demand)
+ die(_("Server does not support --on-demand"));
if (everything_local(args, &ref, sought, nr_sought)) {
packet_flush(fd[1]);
diff --git a/fetch-pack.h b/fetch-pack.h
index c912e3d..16ab8bd 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -29,6 +29,7 @@ struct fetch_pack_args {
unsigned cloning:1;
unsigned update_shallow:1;
unsigned deepen:1;
+ unsigned on_demand:1;
};
/*
diff --git a/shallow.c b/shallow.c
index 11f7dde..a24292b 100644
--- a/shallow.c
+++ b/shallow.c
@@ -45,7 +45,7 @@ int is_repository_shallow(void)
FILE *fp;
char buf[1024];
const char *path = alternate_shallow_file;
-
+ is_shallow = 0; /*** FIXME: markbt temp hack to allow shallow repos with on-demand files ***/
if (is_shallow >= 0)
return is_shallow;
diff --git a/transport.c b/transport.c
index 5828e06..69f8d72 100644
--- a/transport.c
+++ b/transport.c
@@ -160,6 +160,8 @@ static int set_git_option(struct git_transport_options *opts,
} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {
opts->deepen_relative = !!value;
return 0;
+ } else if (!strcmp(name, TRANS_OPT_ON_DEMAND)) {
+ opts->on_demand = !!value;
}
return 1;
}
@@ -223,6 +225,7 @@ static int fetch_refs_via_pack(struct transport *transport,
args.deepen_since = data->options.deepen_since;
args.deepen_not = data->options.deepen_not;
args.deepen_relative = data->options.deepen_relative;
+ args.on_demand = data->options.on_demand;
args.check_self_contained_and_connected =
data->options.check_self_contained_and_connected;
args.cloning = transport->cloning;
diff --git a/transport.h b/transport.h
index bc55715..d4f848b 100644
--- a/transport.h
+++ b/transport.h
@@ -15,6 +15,7 @@ struct git_transport_options {
unsigned self_contained_and_connected : 1;
unsigned update_shallow : 1;
unsigned deepen_relative : 1;
+ unsigned on_demand : 1;
int depth;
const char *deepen_since;
const struct string_list *deepen_not;
@@ -210,6 +211,9 @@ void transport_check_allowed(const char *type);
/* Send push certificates */
#define TRANS_OPT_PUSH_CERT "pushcert"
+/* On-demand clone */
+#define TRANS_OPT_ON_DEMAND "on-demand"
+
/**
* Returns 0 if the option was used, non-zero otherwise. Prints a
* message to stderr if the option is not used.
--
2.7.4
^ permalink raw reply related
* [RFC 1/4] upload-file: Add upload-file command
From: Mark Thomas @ 2017-03-04 19:18 UTC (permalink / raw)
To: git; +Cc: Mark Thomas
In-Reply-To: <20170304191901.9622-1-markbt@efaref.net>
The upload-file command allows a remote to request specific files by sha1.
Signed-off-by: Mark Thomas <markbt@efaref.net>
---
.gitignore | 1 +
Makefile | 2 ++
daemon.c | 6 +++++
upload-file.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 96 insertions(+)
create mode 100644 upload-file.c
diff --git a/.gitignore b/.gitignore
index 833ef3b..c2db9c2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -165,6 +165,7 @@
/git-update-ref
/git-update-server-info
/git-upload-archive
+/git-upload-file
/git-upload-pack
/git-var
/git-verify-commit
diff --git a/Makefile b/Makefile
index 9ec6065..1b84322 100644
--- a/Makefile
+++ b/Makefile
@@ -597,6 +597,7 @@ PROGRAM_OBJS += sh-i18n--envsubst.o
PROGRAM_OBJS += shell.o
PROGRAM_OBJS += show-index.o
PROGRAM_OBJS += upload-pack.o
+PROGRAM_OBJS += upload-file.o
PROGRAM_OBJS += remote-testsvn.o
# Binary suffix, set to .exe for Windows builds
@@ -668,6 +669,7 @@ BINDIR_PROGRAMS_NEED_X += git-upload-pack
BINDIR_PROGRAMS_NEED_X += git-receive-pack
BINDIR_PROGRAMS_NEED_X += git-upload-archive
BINDIR_PROGRAMS_NEED_X += git-shell
+BINDIR_PROGRAMS_NEED_X += git-upload-file
BINDIR_PROGRAMS_NO_X += git-cvsserver
diff --git a/daemon.c b/daemon.c
index 473e6b6..8b5b026 100644
--- a/daemon.c
+++ b/daemon.c
@@ -479,6 +479,11 @@ static int upload_pack(void)
return run_service_command(argv);
}
+static int upload_file(void)
+{
+ const char *argv[] = { "upload-file", ".", NULL };
+ return run_service_command(argv);
+}
static int upload_archive(void)
{
static const char *argv[] = { "upload-archive", ".", NULL };
@@ -494,6 +499,7 @@ static int receive_pack(void)
static struct daemon_service daemon_service[] = {
{ "upload-archive", "uploadarch", upload_archive, 0, 1 },
{ "upload-pack", "uploadpack", upload_pack, 1, 1 },
+ { "upload-file", "uploadfile", upload_file, 1, 1 },
{ "receive-pack", "receivepack", receive_pack, 0, 1 },
};
diff --git a/upload-file.c b/upload-file.c
new file mode 100644
index 0000000..cb2bfe8
--- /dev/null
+++ b/upload-file.c
@@ -0,0 +1,87 @@
+
+#include "cache.h"
+#include "exec_cmd.h"
+#include "parse-options.h"
+#include "pkt-line.h"
+
+static const char * const upload_file_usage[] = {
+ N_("git upload-file [<options>] <dir>"),
+ NULL
+};
+
+
+static void upload_file(void)
+{
+ for (;;) {
+ char *line = packet_read_line(0, NULL);
+ const char *arg;
+ if (!line)
+ break;
+
+ if (skip_prefix(line, "info ", &arg)) {
+ unsigned char sha1[20];
+ void *buffer;
+ enum object_type type;
+ unsigned long size;
+
+ if (get_sha1_hex(arg, sha1))
+ die("invalid sha: %s", arg);
+
+ buffer = read_sha1_file(sha1, &type, &size);
+ if (buffer) {
+ packet_write_fmt(1, "found %s %d %ld\n", sha1_to_hex(sha1), type, size);
+ free(buffer);
+ } else {
+ packet_write_fmt(1, "missing %s\n", sha1_to_hex(sha1));
+ }
+ }
+
+ if (skip_prefix(line, "get ", &arg)) {
+ unsigned char sha1[20];
+ void *buffer;
+ enum object_type type;
+ unsigned long size;
+
+ if (get_sha1_hex(arg, sha1))
+ die("invalid sha: %s", arg);
+
+ buffer = read_sha1_file(sha1, &type, &size);
+ if (buffer) {
+ packet_write_fmt(1, "found %s %d %ld\n", sha1_to_hex(sha1), type, size);
+ write_or_die(1, buffer, size);
+ free(buffer);
+ } else {
+ packet_write_fmt(1, "missing %s\n", sha1_to_hex(sha1));
+ }
+
+ }
+
+ if (!strcmp(line, "end"))
+ break;
+ }
+}
+
+int cmd_main(int argc, const char **argv)
+{
+ const char *dir;
+ struct option options[] = {
+ OPT_END()
+ };
+
+ packet_trace_identity("upload-file");
+
+ argc = parse_options(argc, argv, NULL, options, upload_file_usage, 0);
+
+ if (argc != 1)
+ usage_with_options(upload_file_usage, options);
+
+ setup_path();
+
+ dir = argv[0];
+
+ if (!enter_repo(dir, 0))
+ die("'%s' does not appear to be a git repository", dir);
+
+ upload_file();
+ return 0;
+}
--
2.7.4
^ permalink raw reply related
* [RFC 2/4] on-demand: Fetch missing files from remote
From: Mark Thomas @ 2017-03-04 19:18 UTC (permalink / raw)
To: git; +Cc: Mark Thomas
In-Reply-To: <20170304191901.9622-1-markbt@efaref.net>
If an object (tree, blob, ...) is not found either in the
packs or loose, check if it is available on-demand from
the remote.
Signed-off-by: Mark Thomas <markbt@efaref.net>
---
Makefile | 1 +
cache.h | 3 +-
on_demand.c | 157 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
on_demand.h | 8 ++++
sha1_file.c | 8 +++-
5 files changed, 174 insertions(+), 3 deletions(-)
create mode 100644 on_demand.c
create mode 100644 on_demand.h
diff --git a/Makefile b/Makefile
index 1b84322..fb8ca6c 100644
--- a/Makefile
+++ b/Makefile
@@ -784,6 +784,7 @@ LIB_OBJS += notes-merge.o
LIB_OBJS += notes-utils.o
LIB_OBJS += object.o
LIB_OBJS += oidset.o
+LIB_OBJS += on_demand.o
LIB_OBJS += pack-bitmap.o
LIB_OBJS += pack-bitmap-write.o
LIB_OBJS += pack-check.o
diff --git a/cache.h b/cache.h
index 80b6372..d34af06 100644
--- a/cache.h
+++ b/cache.h
@@ -1730,7 +1730,8 @@ struct object_info {
OI_CACHED,
OI_LOOSE,
OI_PACKED,
- OI_DBCACHED
+ OI_DBCACHED,
+ OI_ONDEMAND
} whence;
union {
/*
diff --git a/on_demand.c b/on_demand.c
new file mode 100644
index 0000000..a0aaf18
--- /dev/null
+++ b/on_demand.c
@@ -0,0 +1,157 @@
+#include "transport.h"
+#include "pkt-line.h"
+#include "remote.h"
+#include "commit.h"
+
+static struct remote *remote = NULL;
+static struct transport *transport = NULL;
+static int fd[2];
+static int connected;
+
+struct trace_key trace_on_demand = TRACE_KEY_INIT(ON_DEMAND);
+
+static void on_demand_cleanup(void)
+{
+ if (connected) {
+ packet_write_fmt(fd[1], "end\n");
+ transport_disconnect(transport);
+ connected = 0;
+ }
+}
+
+static int on_demand_connect(void)
+{
+ if (!connected) {
+ if (!remote)
+ remote = remote_get(NULL);
+ if (remote && !transport)
+ transport = transport_get(remote, NULL);
+ if (!remote || !transport)
+ return 0;
+ if (transport_connect(transport, transport->url, "git-upload-file", fd))
+ return 0;
+ connected = 1;
+ atexit(on_demand_cleanup);
+ }
+ return 1;
+}
+
+void *read_remote_on_demand(const unsigned char *sha1, enum object_type *type,
+ unsigned long *size)
+{
+ const char *line;
+ const char *arg;
+ int line_size;
+
+ if (!on_demand_connect())
+ return NULL;
+
+ packet_write_fmt(fd[1], "get %s\n", sha1_to_hex(sha1));
+
+ line = packet_read_line(fd[0], &line_size);
+
+ if (line_size == 0)
+ return NULL;
+
+ if (skip_prefix(line, "missing ", &arg))
+ return NULL;
+
+ if (skip_prefix(line, "found ", &arg)) {
+ char *end = NULL;
+ void *buffer;
+ unsigned char file_sha1[GIT_SHA1_RAWSZ];
+ enum object_type file_type;
+ unsigned long file_size;
+ ssize_t size_read;
+
+ if (get_sha1_hex(arg, file_sha1))
+ die("git on-demand: protocol error, "
+ "expected to get sha in '%s'", line);
+ arg += GIT_SHA1_HEXSZ;
+
+ file_type = strtol(arg, &end, 0);
+ if (!end || file_type < 0 || file_type >= OBJ_MAX)
+ die("git on-demand: protocol error, "
+ "invalid object type in '%s'", line);
+ arg = end;
+
+ file_size = strtoul(arg, &end, 0);
+ if (!end || *end || file_size > LONG_MAX)
+ die("git on-demand: protocol error, "
+ "invalid file size in '%s'", line);
+
+ buffer = xmalloc(file_size);
+ if (!buffer)
+ die("git on-demand: failed to allocate "
+ "buffer for %ld bytes", file_size);
+
+ size_read = read_in_full(fd[0], buffer, file_size);
+ if (size_read != (ssize_t)file_size)
+ die("git on-demand: protocol error, "
+ "failed to read file data");
+
+ trace_printf_key(&trace_on_demand, "on-demand: fetched %s\n",
+ sha1_to_hex(sha1));
+ *type = file_type;
+ *size = file_size;
+ return buffer;
+ }
+
+ die("git on-demand: protocol error, "
+ "unexpected response: '%s'", line);
+}
+
+int object_info_on_demand(const unsigned char *sha1, struct object_info *oi)
+{
+ const char *line;
+ const char *arg;
+ int line_size;
+
+ if (!on_demand_connect())
+ return -1;
+
+ packet_write_fmt(fd[1], "info %s\n", sha1_to_hex(sha1));
+
+ line = packet_read_line(fd[0], &line_size);
+
+ if (line_size == 0)
+ return -1;
+
+ if (skip_prefix(line, "missing ", &arg))
+ return -1;
+
+ if (skip_prefix(line, "found ", &arg)) {
+ char *end = NULL;
+ unsigned char sha1[GIT_SHA1_RAWSZ];
+ enum object_type file_type;
+ unsigned long file_size;
+
+ if (get_sha1_hex(arg, sha1))
+ die("git on-demand: protocol error, "
+ "expected to get sha in '%s'", line);
+ arg += GIT_SHA1_HEXSZ;
+
+ file_type = strtol(arg, &end, 0);
+ if (!end || file_type < 0 || file_type >= OBJ_MAX)
+ die("git on-demand: protocol error, "
+ "invalid object type in '%s'", line);
+ arg = end;
+
+ file_size = strtoul(arg, &end, 0);
+ if (!end || *end || file_size > LONG_MAX)
+ die("git on-demand: protocol error, "
+ "invalid file size in '%s'", line);
+
+ if (oi->typep)
+ *oi->typep = file_type;
+ if (oi->sizep)
+ *oi->typep = file_size;
+ if (oi->disk_sizep)
+ *oi->disk_sizep = 0;
+ oi->whence = OI_ONDEMAND;
+ return 0;
+ }
+
+ die("git on-demand: protocol error, "
+ "unexpected response: '%s'", line);
+}
diff --git a/on_demand.h b/on_demand.h
new file mode 100644
index 0000000..09a8072
--- /dev/null
+++ b/on_demand.h
@@ -0,0 +1,8 @@
+#ifndef ON_DEMAND_H
+#define ON_DEMAND_H
+
+void *read_remote_on_demand(const unsigned char *sha1, enum object_type *type,
+ unsigned long *size);
+int object_info_on_demand(const unsigned char *sha1, struct object_info *oi);
+
+#endif
diff --git a/sha1_file.c b/sha1_file.c
index 6628f06..510da41 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -27,6 +27,7 @@
#include "list.h"
#include "mergesort.h"
#include "quote.h"
+#include "on_demand.h"
#define SZ_FMT PRIuMAX
static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -2979,7 +2980,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
/* Not a loose object; someone else may have just packed it. */
reprepare_packed_git();
if (!find_pack_entry(real, &e))
- return -1;
+ return object_info_on_demand(real, oi);
}
/*
@@ -3091,7 +3092,10 @@ static void *read_object(const unsigned char *sha1, enum object_type *type,
return buf;
}
reprepare_packed_git();
- return read_packed_sha1(sha1, type, size);
+ buf = read_packed_sha1(sha1, type, size);
+ if (buf)
+ return buf;
+ return read_remote_on_demand(sha1, type, size);
}
/*
--
2.7.4
^ permalink raw reply related
* Re: [PATCH v1] Travis: also test on 32-bit Linux
From: Vegard Nossum @ 2017-03-04 19:49 UTC (permalink / raw)
To: Lars Schneider, Junio C Hamano, allan.x.xavier, Jeff King
Cc: Johannes Schindelin, Git Mailing List
In-Reply-To: <f5f5886a-aaec-7426-ea33-f5d65516348b@oracle.com>
On 04/03/2017 19:08, Vegard Nossum wrote:
> On 04/03/2017 18:23, Lars Schneider wrote:
>> Did Travis find our first 32bit bug? I briefly looked into it
>> and the following new topic on pu seems to cause the issue:
>>
>> http://public-inbox.org/git/20170302172902.16850-1-allan.x.xavier@oracle.com/
>>
>> https://github.com/git/git/commit/aaae0bf787f09ba102f69c3cf85d37e6554ab9fd
>>
>>
>> The "git log" call in the new 4211 test fails with:
>> *** Error in `/usr/src/git/git': malloc: top chunk is corrupt:
>> 0x09ff4a78 ***
>>
>> More output here:
>> https://travis-ci.org/larsxschneider/git/builds/207715343
>
> It looks like it's hitting the bug the patch is supposed to fix.
>
> Are you quite sure it's running the "git" binary that was just built (as
> opposed to e.g. the system binary installed inside the container)?
Nevermind, I can reproduce it locally.
==10205== Invalid write of size 4
==10205== at 0x4031ED2: memcpy (in
/usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==10205== by 0x811C4B0: memcpy (string3.h:53)
==10205== by 0x811C4B0: range_set_copy.isra.7 (line-log.c:46)
==10205== by 0x811C51B: line_log_data_copy_one (line-log.c:628)
==10205== by 0x811C55D: line_log_data_copy (line-log.c:642)
==10205== by 0x811C5E3: add_line_range (line-log.c:708)
==10205== by 0x811D767: line_log_init (line-log.c:745)
==10205== by 0x808B1CD: cmd_log_init_finish (log.c:204)
==10205== by 0x808C9C8: cmd_log_init (log.c:213)
==10205== by 0x808C9C8: cmd_log (log.c:681)
==10205== by 0x804CF14: run_builtin (git.c:373)
==10205== by 0x804CF14: handle_builtin (git.c:574)
==10205== by 0x804D264: run_argv (git.c:626)
==10205== by 0x804D264: cmd_main (git.c:703)
==10205== by 0x804C448: main (common-main.c:43)
==10205== Address 0x4cde9c8 is 0 bytes after a block of size 1,600 alloc'd
==10205== at 0x402D17C: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==10205== by 0x402F370: realloc (in
/usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==10205== by 0x819CC0F: xrealloc (wrapper.c:137)
==10205== by 0x811C1C8: range_set_grow (line-log.c:21)
==10205== by 0x811C499: range_set_init (line-log.c:32)
==10205== by 0x811C499: range_set_copy.isra.7 (line-log.c:45)
==10205== by 0x811C51B: line_log_data_copy_one (line-log.c:628)
==10205== by 0x811C55D: line_log_data_copy (line-log.c:642)
==10205== by 0x811C5E3: add_line_range (line-log.c:708)
==10205== by 0x811D767: line_log_init (line-log.c:745)
==10205== by 0x808B1CD: cmd_log_init_finish (log.c:204)
==10205== by 0x808C9C8: cmd_log_init (log.c:213)
==10205== by 0x808C9C8: cmd_log (log.c:681)
==10205== by 0x804CF14: run_builtin (git.c:373)
==10205== by 0x804CF14: handle_builtin (git.c:574)
At a glance, looks like range_set_copy() is using
sizeof(struct range_set) == 12, but
range_set_init/range_set_grow/ALLOC_GROW/REALLOC_ARRAY is using
sizeof(rs->range) == 8.
Vegard
^ permalink raw reply
* Re: [PATCH v1 1/1] git diff --quiet exits with 1 on clean tree with CRLF conversions
From: Junio C Hamano @ 2017-03-04 19:59 UTC (permalink / raw)
To: Torsten Bögershausen; +Cc: Mike Crowe, git, Jeff King
In-Reply-To: <9c9eeb35-e1c1-ec46-1d85-ef6a05886880@web.de>
Torsten Bögershausen <tboegi@web.de> writes:
>>> enum safe_crlf crlf_warn = (safe_crlf == SAFE_CRLF_FAIL
>>> ? SAFE_CRLF_WARN
>>> : safe_crlf);
>>> + if (size_only)
>>> + crlf_warn = SAFE_CRLF_FALSE;
>>
>> If you were to go this route, it may be sufficient to change its
>> initialization from WARN to FALSE _unconditionally_, because this
>> function uses the convert_to_git() only to _show_ the differences by
>> computing canonical form out of working tree contents, and the
>> conversion is not done to _write_ into object database to create a
>> new object.
>
> Hm, since when (is it not used) ?
Since forever, but my statement above said "this function", which may
have confused you, where it could have said diff_populate_filespec().
Surely it is possible for somebody to diff_populate_filespec(s, 0)
and then call hash_sha1_file(s->data, s->size, "blob", ...) to write
the data into the object database to create a new object. But that
sounds really crazy, no?
> The SAFE_CRLF_FAIL was converted into WARN here:
> commit 5430bb283b478991a979437a79e10dcbb6f20e28
> Author: Junio C Hamano <gitster@pobox.com>
> Date: Mon Jun 24 14:35:04 2013 -0700
>
> diff: demote core.safecrlf=true to core.safecrlf=warn
Yes.
> Does this all means that, looking back, 5430bb283b478991 could have been more
> aggressive and could have used SAFE_CRLF_FALSE ?
That is pretty much the statement, to which you said "since when",
suspects.
> And we can do this change now?
I am not sure. The conversion the safe-crlf code does is unsafe and
it is a disservice to users not to warn whenever we notice they are
risking information loss. Maybe time they run "git diff" is not a
good time to warn, as they may not be actually adding the file
as-is, but if warning against information loss at "git diff" time is
important enough, the I think that should not be squelched by the
"--quiet" option, which is about "do not show the patch text
output". It should not be taken as "do not diagnose any errors".
^ permalink raw reply
* Re: [PATCH v1] Travis: also test on 32-bit Linux
From: Vegard Nossum @ 2017-03-04 20:08 UTC (permalink / raw)
To: Lars Schneider, Junio C Hamano, allan.x.xavier, Jeff King
Cc: Johannes Schindelin, Git Mailing List
In-Reply-To: <af31ef46-bd0c-c3f2-5a1e-7d97da6ec9a0@oracle.com>
[-- Attachment #1: Type: text/plain, Size: 1114 bytes --]
On 04/03/2017 20:49, Vegard Nossum wrote:
> On 04/03/2017 19:08, Vegard Nossum wrote:
>> On 04/03/2017 18:23, Lars Schneider wrote:
>>> Did Travis find our first 32bit bug? I briefly looked into it
>>> and the following new topic on pu seems to cause the issue:
>>>
>>> http://public-inbox.org/git/20170302172902.16850-1-allan.x.xavier@oracle.com/
>>>
>>>
>>> https://github.com/git/git/commit/aaae0bf787f09ba102f69c3cf85d37e6554ab9fd
>>>
>>>
>>>
>>> The "git log" call in the new 4211 test fails with:
>>> *** Error in `/usr/src/git/git': malloc: top chunk is corrupt:
>>> 0x09ff4a78 ***
>>>
>>> More output here:
>>> https://travis-ci.org/larsxschneider/git/builds/207715343
[...]
> At a glance, looks like range_set_copy() is using
> sizeof(struct range_set) == 12, but
> range_set_init/range_set_grow/ALLOC_GROW/REALLOC_ARRAY is using
> sizeof(rs->range) == 8.
Attached patch seems to fix it -- basically, range_set_copy() is trying
to copy more than it should. It was uncovered with the test case from
Allan's commit because it's creating enough ranges to overflow the
initial allocation on 32-bit.
Vegard
[-- Attachment #2: range_set_copy.patch --]
[-- Type: text/x-patch, Size: 511 bytes --]
diff --git a/line-log.c b/line-log.c
index 951029665..cb0dc1110 100644
--- a/line-log.c
+++ b/line-log.c
@@ -43,7 +43,7 @@ void range_set_release(struct range_set *rs)
static void range_set_copy(struct range_set *dst, struct range_set *src)
{
range_set_init(dst, src->nr);
- memcpy(dst->ranges, src->ranges, src->nr*sizeof(struct range_set));
+ memcpy(dst->ranges, src->ranges, src->nr*sizeof(struct range));
dst->nr = src->nr;
}
static void range_set_move(struct range_set *dst, struct range_set *src)
^ permalink raw reply related
* Re: [PATCH 3/3] Remove outdated info in difftool manpage
From: David Aguilar @ 2017-03-04 20:46 UTC (permalink / raw)
To: Denton Liu; +Cc: Johannes Schindelin, git
In-Reply-To: <20170303212836.GB13790@arch-attack.localdomain>
On Fri, Mar 03, 2017 at 01:28:36PM -0800, Denton Liu wrote:
> On Fri, Mar 03, 2017 at 04:46:36PM +0100, Johannes Schindelin wrote:
> > Hi Denton (or should I address you as Liu?),
> Denton is fine, thanks.
> >
> > On Fri, 3 Mar 2017, Denton Liu wrote:
> >
> > > When difftool was rewritten in C, it removed the capability to read
> > > fallback configs from mergetool. This changes the documentation to
> > > reflect this.
> >
> > Thanks for pointing that out. But that is probably an oversight on my
> > part, not an intentional change...
> Do you expect to be submitting a patch for this soon? Or, if not, would
> it be fine if I went ahead and did it?
Thanks for spotting this. It'd be good to fix this so
I'm sure no one would mind if you submitted a patch ;-)
I'd be happy to test your patch if you have one.
--
David
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox