* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed
From: Jeff King @ 2016-12-14 21:12 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder
In-Reply-To: <20161214203349.6ym3v2ny2gonovx2@sigill.intra.peff.net>
On Wed, Dec 14, 2016 at 03:33:49PM -0500, Jeff King wrote:
> One other "simple" fix is at the moment we parse http-alternates, to
> parse the URL ourself and check the policy. Like:
> [...]
> I may have convinced myself this is a reasonable approach.
So here it is in patch form, with a test.
I also took a look at how bad it would be to plumb through the "this is
an alternate" flag. And it's actually a little nasty, because the
http-walker isn't calling get_active_slot() itself. It's relying on
http_get_file() and other wrappers. The recent http_get_options makes
this slightly less terrible, but I'd still rather avoid infecting the
general http code that is used for the smart-http code paths.
So I think this patch on top of your series, plus the other minor fixes
we've discussed, the topic should be ready for 'next'.
-- >8 --
Subject: http: respect protocol.*.allow=user for http-alternates
The http-walker may fetch the http-alternates (or
alternates) file from a remote in order to find more
objects. This should count as a "not from the user" use of
the protocol. But because we implement the redirection
ourselves and feed the new URL to curl, it will use the
CURLOPT_PROTOCOLS rules, not the more restrictive
CURLOPT_REDIR_PROTOCOLS.
The ideal solution would be for each curl request we make to
know whether or not is directly from the user or part of an
alternates redirect, and then set CURLOPT_PROTOCOLS as
appropriate. However, that would require plumbing that
information through all of the various layers of the http
code.
Instead, let's check the protocol at the source: when we are
parsing the remote http-alternates file. The only downside
is that if there's any mismatch between what protocol we
think it is versus what curl thinks it is, it could violate
the policy.
To address this, we'll make the parsing err on the picky
side, and only allow protocols that it can parse
definitively. So for example, you can't elude the "http"
policy by asking for "HTTP://", even though curl might
handle it; we would reject it as unknown. The only unsafe
case would be if you have a URL that starts with "http://"
but curl interprets as another protocol. That seems like an
unlikely failure mode (and we are still protected by our
base CURLOPT_PROTOCOL setting, so the worst you could do is
trigger one of https, ftp, or ftps).
Signed-off-by: Jeff King <peff@peff.net>
---
I actually do reject "HTTP://" here, though I suspect curl does take it.
I notice that you cannot ask to clone "HTTP://..." in the first place,
as our remote-helper interface is case sensitive (or maybe it would even
respect "HTTP://" on a case-insensitive file system!).
Possibly we should be more consistent about down-casing protocols when
comparing them, but I'm not sure if anybody actually cares in practice.
http-walker.c | 52 ++++++++++++++++++++++++++++++++++++----------
t/t5550-http-fetch-dumb.sh | 10 +++++++++
2 files changed, 51 insertions(+), 11 deletions(-)
diff --git a/http-walker.c b/http-walker.c
index c2f81cd6a..b34b6ace7 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -3,6 +3,7 @@
#include "walker.h"
#include "http.h"
#include "list.h"
+#include "transport.h"
struct alt_base {
char *base;
@@ -160,6 +161,32 @@ static void prefetch(struct walker *walker, unsigned char *sha1)
#endif
}
+static int is_alternate_allowed(const char *url)
+{
+ const char *protocols[] = {
+ "http", "https", "ftp", "ftps"
+ };
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(protocols); i++) {
+ const char *end;
+ if (skip_prefix(url, protocols[i], &end) &&
+ starts_with(end, "://"))
+ break;
+ }
+
+ if (i >= ARRAY_SIZE(protocols)) {
+ warning("ignoring alternate with unknown protocol: %s", url);
+ return 0;
+ }
+ if (!is_transport_allowed(protocols[i], 0)) {
+ warning("ignoring alternate with restricted protocol: %s", url);
+ return 0;
+ }
+
+ return 1;
+}
+
static void process_alternates_response(void *callback_data)
{
struct alternates_request *alt_req =
@@ -274,17 +301,20 @@ static void process_alternates_response(void *callback_data)
struct strbuf target = STRBUF_INIT;
strbuf_add(&target, base, serverlen);
strbuf_add(&target, data + i, posn - i - 7);
- warning("adding alternate object store: %s",
- target.buf);
- newalt = xmalloc(sizeof(*newalt));
- newalt->next = NULL;
- newalt->base = strbuf_detach(&target, NULL);
- newalt->got_indices = 0;
- newalt->packs = NULL;
-
- while (tail->next != NULL)
- tail = tail->next;
- tail->next = newalt;
+
+ if (is_alternate_allowed(target.buf)) {
+ warning("adding alternate object store: %s",
+ target.buf);
+ newalt = xmalloc(sizeof(*newalt));
+ newalt->next = NULL;
+ newalt->base = strbuf_detach(&target, NULL);
+ newalt->got_indices = 0;
+ newalt->packs = NULL;
+
+ while (tail->next != NULL)
+ tail = tail->next;
+ tail->next = newalt;
+ }
}
}
i = posn + 1;
diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh
index 22011f0b6..c0ee29c65 100755
--- a/t/t5550-http-fetch-dumb.sh
+++ b/t/t5550-http-fetch-dumb.sh
@@ -360,5 +360,15 @@ test_expect_success 'http-alternates cannot point at funny protocols' '
clone "$HTTPD_URL/dumb/evil.git" evil-file
'
+test_expect_success 'http-alternates triggers not-from-user protocol check' '
+ echo "$HTTPD_URL/dumb/victim.git/objects" \
+ >"$evil/objects/info/http-alternates" &&
+ test_config_global http.followRedirects true &&
+ test_must_fail git -c protocol.http.allow=user \
+ clone $HTTPD_URL/dumb/evil.git evil-user &&
+ git -c protocol.http.allow=always \
+ clone $HTTPD_URL/dumb/evil.git evil-user
+'
+
stop_httpd
test_done
--
2.11.0.341.g202cd3142
^ permalink raw reply related
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed
From: Brandon Williams @ 2016-12-14 20:50 UTC (permalink / raw)
To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder
In-Reply-To: <20161214204102.hwjb3i4aaxf3oigq@sigill.intra.peff.net>
On 12/14, Jeff King wrote:
> On Wed, Dec 14, 2016 at 12:37:52PM -0800, Brandon Williams wrote:
>
> > Naively looking at the code (and your longer suggestion), is there a
> > reason why we couldn't simply have http-walker set CURLOPT_PROTOCOLS
> > with get_curl_allowed_protocols(0) in the fetch_alternates() function?
> > That way we just override the CURLOPT_PROTOCOLS value when alternates
> > are involved.
>
> No, because we may have many curl handles (especially for the
> http-walker, which wants to fetch several objects simultaneously), and
> they get recycled as needed for many requests.
>
> So setting a restriction there on slot->curl will only cover the one
> handle, and miss other ones which may be used later (and likewise, that
> one handle with the restriction may get recycled and used for a
> non-alternate fetch, and would be unnecessarily restrictive).
>
> That's why any curl-level settings have to happen when we call
> get_active_slot(), since that's when we know what we're actually using
> the handle for.
Fair enough, I figured there may be some reuse happening with the curl
handles but didn't do enough digging to discover that myself. Thanks :)
--
Brandon Williams
^ permalink raw reply
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed
From: Jeff King @ 2016-12-14 20:41 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder
In-Reply-To: <20161214203752.GD20063@google.com>
On Wed, Dec 14, 2016 at 12:37:52PM -0800, Brandon Williams wrote:
> Naively looking at the code (and your longer suggestion), is there a
> reason why we couldn't simply have http-walker set CURLOPT_PROTOCOLS
> with get_curl_allowed_protocols(0) in the fetch_alternates() function?
> That way we just override the CURLOPT_PROTOCOLS value when alternates
> are involved.
No, because we may have many curl handles (especially for the
http-walker, which wants to fetch several objects simultaneously), and
they get recycled as needed for many requests.
So setting a restriction there on slot->curl will only cover the one
handle, and miss other ones which may be used later (and likewise, that
one handle with the restriction may get recycled and used for a
non-alternate fetch, and would be unnecessarily restrictive).
That's why any curl-level settings have to happen when we call
get_active_slot(), since that's when we know what we're actually using
the handle for.
-Peff
^ permalink raw reply
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed
From: Brandon Williams @ 2016-12-14 20:37 UTC (permalink / raw)
To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder
In-Reply-To: <20161214201323.GC20063@google.com>
On 12/14, Brandon Williams wrote:
> On 12/14, Jeff King wrote:
> > On Tue, Dec 13, 2016 at 05:40:37PM -0800, Brandon Williams wrote:
> >
> > > Add the from_user parameter to the 'is_transport_allowed' function.
> > > This allows callers to query if a transport protocol is allowed, given
> > > that the caller knows that the protocol is coming from the user (1) or
> > > not from the user (0) such as redirects in libcurl. If unknown a -1
> > > should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER`
> > > to determine if the protocol came from the user.
> >
> > I think your commit message is upside-down with respect to the purpose
> > of the patch. The end goal we want is for http to distinguish between
> > protocol restrictions for redirects versus initial requests. The rest is
> > an implementation detail. It's definitely still worth discussing that
> > implementation detail (though I think your in-code comments may be
> > sufficient), but I don't see the rationale discussed here at all.
>
> I'll fix the commit message to better discuss the reasoning behind the
> change.
>
> > > Signed-off-by: Brandon Williams <bmwill@google.com>
> > > ---
> > > http.c | 14 +++++++-------
> > > transport.c | 8 +++++---
> > > transport.h | 13 ++++++++++---
> > > 3 files changed, 22 insertions(+), 13 deletions(-)
> >
> > I'm trying to think of a way to test this. I guess the case we are
> > covering here is when a server redirects, but the protocol is only
> > allowed from the user. So:
> >
> > diff --git a/t/t5812-proto-disable-http.sh b/t/t5812-proto-disable-http.sh
> > index 044cc152f..d911afd24 100755
> > --- a/t/t5812-proto-disable-http.sh
> > +++ b/t/t5812-proto-disable-http.sh
> > @@ -30,5 +30,12 @@ test_expect_success 'curl limits redirects' '
> > test_must_fail git clone "$HTTPD_URL/loop-redir/smart/repo.git"
> > '
> >
> > +test_expect_success 'http can be limited to from-user' '
> > + git -c protocol.http.allow=user \
> > + clone "$HTTPD_URL/smart/repo.git" plain.git &&
> > + test_must_fail git -c protocol.http.allow=user \
> > + clone "$HTTPD_URL/smart-redir-perm/repo.git" redir.git
> > +'
> > +
> > stop_httpd
> > test_done
> >
> > It's an oddball configuration, and you'd probably just set
> > http.followRedirects=false in practice, but it does correctly check this
> > case.
>
> K I'll add this in as a test.
>
> > > @@ -588,9 +588,9 @@ static CURL *get_curl_handle(void)
> > > #endif
> > > #if LIBCURL_VERSION_NUM >= 0x071304
> > > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS,
> > > - get_curl_allowed_protocols());
> > > + get_curl_allowed_protocols(0));
> > > curl_easy_setopt(result, CURLOPT_PROTOCOLS,
> > > - get_curl_allowed_protocols());
> > > + get_curl_allowed_protocols(-1));
> >
> > This covers internal redirects done by libcurl, but not the dumb-walker
> > http-alternates nonsense. We have to feed the URL from http-alternates
> > back to curl ourselves, so it uses CURLOPT_PROTOCOLS even though it
> > should count as "not from the user".
> >
> > To fix that, I think we'd need something like:
> >
> > - get_curl_handle() stops setting these options, as it is done only
> > once when the curl handle is initialized. Instead, the protocol
> > restrictions should go into get_active_slot(), which is called for
> > each request. The values set would remain the same, and be the
> > baseline.
> >
> > - the http-walker.c code would need to know when it's requesting from
> > the base URL, and when it's an alternate. I think this would depend
> > on the position of the "alt" in in the linked list it keeps.
> >
> > - when requesting from an alternate, http-walker would set
> > CURLOPT_PROTOCOLS with get_curl_allowed_protocols(0)
> >
> > I have to admit that it sounds like a fair bit of work for a pretty
> > obscure case. You'd have to:
> >
> > 1. Turn http.allowRedirects to "true", to allow redirects even for
> > non-initial contact.
> >
> > 2. Turn one of protocol.{http,https,ftp,ftps}.allow to "user" to
> > restrict it from being used in a redirect.
> >
> > I'm tempted to punt on it and just do:
> >
> > if (http_follow_config == HTTP_FOLLOW_ALWAYS &&
> > get_curl_allowed_protocols(0) != get_curl_allowed_protocols(-1))
> > die("user-only protocol restrictions not implemented for http-alternates");
> >
> > which errs on the safe side. We could even shove that down into the case
> > where we actually see some alternates, like:
> >
> > diff --git a/http-walker.c b/http-walker.c
> > index c2f81cd6a..5bcc850b1 100644
> > --- a/http-walker.c
> > +++ b/http-walker.c
> > @@ -160,6 +160,12 @@ static void prefetch(struct walker *walker, unsigned char *sha1)
> > #endif
> > }
> >
> > +static void check_alternates_protocol_restrictions(void)
> > +{
> > + if (get_curl_allowed_protocols(0) != get_curl_allowed_protocol(-1))
> > + die("user-only protocol restrictions not implemented for http alternates");
> > +}
> > +
> > static void process_alternates_response(void *callback_data)
> > {
> > struct alternates_request *alt_req =
> > @@ -272,6 +278,7 @@ static void process_alternates_response(void *callback_data)
> > /* skip "objects\n" at end */
> > if (okay) {
> > struct strbuf target = STRBUF_INIT;
> > + check_alternates_protocol_restrictions();
> > strbuf_add(&target, base, serverlen);
> > strbuf_add(&target, data + i, posn - i - 7);
> > warning("adding alternate object store: %s",
> >
> > I find it unlikely that anybody would ever care, but at least we'd do
> > the safe thing. I dunno. Maybe I am just being lazy.
>
> Well, that's unfortunate! It does sound like a more full-proof solution
> to these dumb http alternates could be involved. I don't think your
> simple "lazy" solution may be enough to not just die by default.
>
> By default ftp/ftps will have a policy of "user only" which means they
> will be set by the call to get_curl_allowed_protocol(-1) but not set by
> get_curl_allowed_protocol(0). This would result in the call to
> check_alternates_protocol_restrictions failing all the time unless the
> user explicitly sets ftp/ftps to "always" or "never". If that is the
> desired behavior then your proposed solution would be fine, otherwise we
> may have to do the more involved approach.
Naively looking at the code (and your longer suggestion), is there a
reason why we couldn't simply have http-walker set CURLOPT_PROTOCOLS
with get_curl_allowed_protocols(0) in the fetch_alternates() function?
That way we just override the CURLOPT_PROTOCOLS value when alternates
are involved.
Like so:
diff --git a/http-walker.c b/http-walker.c
index c2f81cd..b284cec 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -339,6 +339,8 @@ static void fetch_alternates(struct walker *walker, const char *base)
curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);
curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
curl_easy_setopt(slot->curl, CURLOPT_URL, url.buf);
+ curl_easy_setopt(slot->curl, CURLOPT_PROTOCOLS,
+ get_curl_allowed_protocols(0));
alt_req.base = base;
alt_req.url = &url;
--
Brandon Williams
^ permalink raw reply related
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed
From: Jeff King @ 2016-12-14 20:33 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder
In-Reply-To: <20161214201323.GC20063@google.com>
On Wed, Dec 14, 2016 at 12:13:23PM -0800, Brandon Williams wrote:
> > I find it unlikely that anybody would ever care, but at least we'd do
> > the safe thing. I dunno. Maybe I am just being lazy.
>
> Well, that's unfortunate! It does sound like a more full-proof solution
> to these dumb http alternates could be involved. I don't think your
> simple "lazy" solution may be enough to not just die by default.
>
> By default ftp/ftps will have a policy of "user only" which means they
> will be set by the call to get_curl_allowed_protocol(-1) but not set by
> get_curl_allowed_protocol(0). This would result in the call to
> check_alternates_protocol_restrictions failing all the time unless the
> user explicitly sets ftp/ftps to "always" or "never". If that is the
> desired behavior then your proposed solution would be fine, otherwise we
> may have to do the more involved approach.
Oh, hrm, you're right. I was definitely meaning for this to kick in only
when you had explicitly configured a protocol to "user" (they'd still
need to enable redirects, but that's much more likely).
Arguably ftp should be on the "safe" list, since it's implemented via
the exact same code as https. That list comes originally from 33cfccbbf
(submodule: allow only certain protocols for submodule fetches,
2015-09-16), but that was just based on what I thought was reasonable at
the time.
I find it hard to believe anybody actually uses git-over-ftp these days,
let alone as a protocol redirect via http-alternates. But AFAIK it does
work.
One other "simple" fix is at the moment we parse http-alternates, to
parse the URL ourself and check the policy. Like:
const char *protocols[] = {
"http", "https", "ftp", "ftps"
};
for (i = 0; i < ARRAY_SIZE(protocols); i++) {
const char *end;
if (skip_prefix(url, protocols[i], end) && starts_with(end, "://"))
break;
}
if (i >= ARRAY_SIZE(protocols))
warning("ignoring alternate with unknown protocol: %s", url);
else if (!is_transport_allowed(protocols[i], 0))
warning("ignoring alternate with restricted protocol: %s", url);
else {
/* actually set up the alt struct */
}
That keeps the logic nicely contained. The downside is that if our
interpretation of the URL is ever different than curl's, it could create
a security problem. The bit above seems fairly foolproof, though,
because it functions as a whitelist. So it doesn't matter if you can
trigger an http request via curl with exotic syntax; it matters whether
curl will trigger anything _besides_ an http syntax if the string starts
with "http://". Which seems unlikely.
I may have convinced myself this is a reasonable approach.
-Peff
^ permalink raw reply
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed
From: Brandon Williams @ 2016-12-14 20:13 UTC (permalink / raw)
To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder
In-Reply-To: <20161214164050.uxk434kzhw6au4c2@sigill.intra.peff.net>
On 12/14, Jeff King wrote:
> On Tue, Dec 13, 2016 at 05:40:37PM -0800, Brandon Williams wrote:
>
> > Add the from_user parameter to the 'is_transport_allowed' function.
> > This allows callers to query if a transport protocol is allowed, given
> > that the caller knows that the protocol is coming from the user (1) or
> > not from the user (0) such as redirects in libcurl. If unknown a -1
> > should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER`
> > to determine if the protocol came from the user.
>
> I think your commit message is upside-down with respect to the purpose
> of the patch. The end goal we want is for http to distinguish between
> protocol restrictions for redirects versus initial requests. The rest is
> an implementation detail. It's definitely still worth discussing that
> implementation detail (though I think your in-code comments may be
> sufficient), but I don't see the rationale discussed here at all.
I'll fix the commit message to better discuss the reasoning behind the
change.
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> > http.c | 14 +++++++-------
> > transport.c | 8 +++++---
> > transport.h | 13 ++++++++++---
> > 3 files changed, 22 insertions(+), 13 deletions(-)
>
> I'm trying to think of a way to test this. I guess the case we are
> covering here is when a server redirects, but the protocol is only
> allowed from the user. So:
>
> diff --git a/t/t5812-proto-disable-http.sh b/t/t5812-proto-disable-http.sh
> index 044cc152f..d911afd24 100755
> --- a/t/t5812-proto-disable-http.sh
> +++ b/t/t5812-proto-disable-http.sh
> @@ -30,5 +30,12 @@ test_expect_success 'curl limits redirects' '
> test_must_fail git clone "$HTTPD_URL/loop-redir/smart/repo.git"
> '
>
> +test_expect_success 'http can be limited to from-user' '
> + git -c protocol.http.allow=user \
> + clone "$HTTPD_URL/smart/repo.git" plain.git &&
> + test_must_fail git -c protocol.http.allow=user \
> + clone "$HTTPD_URL/smart-redir-perm/repo.git" redir.git
> +'
> +
> stop_httpd
> test_done
>
> It's an oddball configuration, and you'd probably just set
> http.followRedirects=false in practice, but it does correctly check this
> case.
K I'll add this in as a test.
> > @@ -588,9 +588,9 @@ static CURL *get_curl_handle(void)
> > #endif
> > #if LIBCURL_VERSION_NUM >= 0x071304
> > curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS,
> > - get_curl_allowed_protocols());
> > + get_curl_allowed_protocols(0));
> > curl_easy_setopt(result, CURLOPT_PROTOCOLS,
> > - get_curl_allowed_protocols());
> > + get_curl_allowed_protocols(-1));
>
> This covers internal redirects done by libcurl, but not the dumb-walker
> http-alternates nonsense. We have to feed the URL from http-alternates
> back to curl ourselves, so it uses CURLOPT_PROTOCOLS even though it
> should count as "not from the user".
>
> To fix that, I think we'd need something like:
>
> - get_curl_handle() stops setting these options, as it is done only
> once when the curl handle is initialized. Instead, the protocol
> restrictions should go into get_active_slot(), which is called for
> each request. The values set would remain the same, and be the
> baseline.
>
> - the http-walker.c code would need to know when it's requesting from
> the base URL, and when it's an alternate. I think this would depend
> on the position of the "alt" in in the linked list it keeps.
>
> - when requesting from an alternate, http-walker would set
> CURLOPT_PROTOCOLS with get_curl_allowed_protocols(0)
>
> I have to admit that it sounds like a fair bit of work for a pretty
> obscure case. You'd have to:
>
> 1. Turn http.allowRedirects to "true", to allow redirects even for
> non-initial contact.
>
> 2. Turn one of protocol.{http,https,ftp,ftps}.allow to "user" to
> restrict it from being used in a redirect.
>
> I'm tempted to punt on it and just do:
>
> if (http_follow_config == HTTP_FOLLOW_ALWAYS &&
> get_curl_allowed_protocols(0) != get_curl_allowed_protocols(-1))
> die("user-only protocol restrictions not implemented for http-alternates");
>
> which errs on the safe side. We could even shove that down into the case
> where we actually see some alternates, like:
>
> diff --git a/http-walker.c b/http-walker.c
> index c2f81cd6a..5bcc850b1 100644
> --- a/http-walker.c
> +++ b/http-walker.c
> @@ -160,6 +160,12 @@ static void prefetch(struct walker *walker, unsigned char *sha1)
> #endif
> }
>
> +static void check_alternates_protocol_restrictions(void)
> +{
> + if (get_curl_allowed_protocols(0) != get_curl_allowed_protocol(-1))
> + die("user-only protocol restrictions not implemented for http alternates");
> +}
> +
> static void process_alternates_response(void *callback_data)
> {
> struct alternates_request *alt_req =
> @@ -272,6 +278,7 @@ static void process_alternates_response(void *callback_data)
> /* skip "objects\n" at end */
> if (okay) {
> struct strbuf target = STRBUF_INIT;
> + check_alternates_protocol_restrictions();
> strbuf_add(&target, base, serverlen);
> strbuf_add(&target, data + i, posn - i - 7);
> warning("adding alternate object store: %s",
>
> I find it unlikely that anybody would ever care, but at least we'd do
> the safe thing. I dunno. Maybe I am just being lazy.
Well, that's unfortunate! It does sound like a more full-proof solution
to these dumb http alternates could be involved. I don't think your
simple "lazy" solution may be enough to not just die by default.
By default ftp/ftps will have a policy of "user only" which means they
will be set by the call to get_curl_allowed_protocol(-1) but not set by
get_curl_allowed_protocol(0). This would result in the call to
check_alternates_protocol_restrictions failing all the time unless the
user explicitly sets ftp/ftps to "always" or "never". If that is the
desired behavior then your proposed solution would be fine, otherwise we
may have to do the more involved approach.
--
Brandon Williams
^ permalink raw reply
* [PATCH v2] fix pushing to //server/share/dir on Windows
From: Johannes Sixt @ 2016-12-14 19:37 UTC (permalink / raw)
To: Jeff King; +Cc: Johannes Schindelin, Git Mailing List
In-Reply-To: <20161214173034.inbyakdykjv5j7ua@sigill.intra.peff.net>
normalize_path_copy() is not prepared to keep the double-slash of a
//server/share/dir kind of path, but treats it like a regular POSIX
style path and transforms it to /server/share/dir.
The bug manifests when 'git push //server/share/dir master' is run,
because tmp_objdir_add_as_alternate() uses the path in normalized
form when it registers the quarantine object database via
link_alt_odb_entries(). Needless to say that the directory cannot be
accessed using the wrongly normalized path.
Fix it by skipping all of the root part, not just a potential drive
prefix. offset_1st_component takes care of this, see the
implementation in compat/mingw.c::mingw_offset_1st_component().
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
---
Am 14.12.2016 um 18:30 schrieb Jeff King:
> Would it be reasonable to
> write:
>
> /* Copy initial part of absolute path, converting separators on Windows */
> const char *end = src + offset_1st_component(src);
> while (src < end) {
> char c = *src++;
> if (c == '\\')
> c = '/';
> *dst++ = c;
> }
Makes a lot of sense! I haven't had an opportunity, though, to test
on Windows.
> ? I'm not sure if it's wrong to convert backslashes in that first
> component or not (but certainly we were before). I don't think we'd need
> is_dir_sep() in that "if()", because we can leave slashes as-is. But
> maybe it would make the code easier to read.
is_dir_sep() is preferable, IMO.
I also changed the commit message and subject line slightly.
path.c | 23 ++++++++++++++---------
1 file changed, 14 insertions(+), 9 deletions(-)
diff --git a/path.c b/path.c
index 52d889c88e..efcedafba6 100644
--- a/path.c
+++ b/path.c
@@ -991,7 +991,7 @@ const char *remove_leading_path(const char *in, const char *prefix)
*
* Performs the following normalizations on src, storing the result in dst:
* - Ensures that components are separated by '/' (Windows only)
- * - Squashes sequences of '/'.
+ * - Squashes sequences of '/' except "//server/share" on Windows
* - Removes "." components.
* - Removes ".." components, and the components the precede them.
* Returns failure (non-zero) if a ".." component appears as first path
@@ -1014,17 +1014,22 @@ const char *remove_leading_path(const char *in, const char *prefix)
int normalize_path_copy_len(char *dst, const char *src, int *prefix_len)
{
char *dst0;
- int i;
+ const char *end;
- for (i = has_dos_drive_prefix(src); i > 0; i--)
- *dst++ = *src++;
+ /*
+ * Copy initial part of absolute path: "/", "C:/", "//server/share/".
+ */
+ end = src + offset_1st_component(src);
+ while (src < end) {
+ char c = *src++;
+ if (is_dir_sep(c))
+ c = '/';
+ *dst++ = c;
+ }
dst0 = dst;
- if (is_dir_sep(*src)) {
- *dst++ = '/';
- while (is_dir_sep(*src))
- src++;
- }
+ while (is_dir_sep(*src))
+ src++;
for (;;) {
char c = *src;
--
2.11.0.79.g263f27a
^ permalink raw reply related
* Re: [PATCH v2 01/34] sequencer: support a new action: 'interactive rebase'
From: Junio C Hamano @ 2016-12-14 19:29 UTC (permalink / raw)
To: Johannes Schindelin, Stephan Beyer; +Cc: git, Kevin Daudt, Dennis Kaarsemaker
In-Reply-To: <297140020a7312af03136848dcdd0353ee3abdfe.1481642927.git.johannes.schindelin@gmx.de>
Johannes Schindelin <johannes.schindelin@gmx.de> writes:
> -/* We will introduce the 'interactive rebase' mode later */
> static inline int is_rebase_i(const struct replay_opts *opts)
> {
> - return 0;
> + return opts->action == REPLAY_INTERACTIVE_REBASE;
> }
>
> static const char *get_dir(const struct replay_opts *opts)
> {
> + if (is_rebase_i(opts))
> + return rebase_path();
> return git_path_seq_dir();
> }
This obviously makes the assumption made by 39784cd362 ("sequencer:
remove useless get_dir() function", 2016-12-07) invalid, where the
"remove useless" thought that the callers of this function wants a
single answer that does not depend on opts.
I'll revert that commit from the sb/sequencer-abort-safety topic (as
the topic is in 'next' already) to keep this one. Please holler if
that is a mistaken decision.
Thanks.
^ permalink raw reply
* Re: [PATCH v2 28/34] run_command_opt(): optionally hide stderr when the command succeeds
From: Johannes Sixt @ 2016-12-14 19:17 UTC (permalink / raw)
To: Jeff King
Cc: Johannes Schindelin, git, Junio C Hamano, Kevin Daudt,
Dennis Kaarsemaker
In-Reply-To: <20161214130640.ginadvry7wor3tkc@sigill.intra.peff.net>
Am 14.12.2016 um 14:06 schrieb Jeff King:
> On Wed, Dec 14, 2016 at 07:53:23AM -0500, Jeff King wrote:
>
>> On Wed, Dec 14, 2016 at 09:34:20AM +0100, Johannes Sixt wrote:
>>
>>> I wanted to see what it would look like if we make it the caller's
>>> responsibility to throw away stderr. The patch is below, as fixup
>>> of patch 29/34. The change is gross, but the end result is not that
>>> bad, though not really a delightful read, either, mostly due to the
>>> strange cleanup semantics of the start_command/finish_command combo,
>>> so... I dunno.
The cleanup semantics of start_command and finish_command are not that
strange as I thought first. I just hadn't looked well enough.
>>
>> I don't have a strong opinion on the patches under discussion, but here
>> are a few pointers on the run-command interface:
>> [...]
>
> And here is a patch representing my suggestions, on top of yours. Not
> tested beyond "make test".
Thank you, that looks way better.
If there is agreement that this approach is preferable, I think we can
have patches on top of the series; they would be orthogonal and do not
have to take hostage of it. (And it looks like I won't be able to follow
up until later this week[end].)
-- Hannes
^ permalink raw reply
* Re: [PATCH v7 00/16] Mark strings in Perl scripts for translation
From: Junio C Hamano @ 2016-12-14 19:02 UTC (permalink / raw)
To: Vasco Almeida
Cc: git, Jiang Xin, Ævar Arnfjörð Bjarmason,
Jean-Noël AVILA, Jakub Narębski, David Aguilar
In-Reply-To: <20161214125439.8822-1-vascomalmeida@sapo.pt>
Vasco Almeida <vascomalmeida@sapo.pt> writes:
> Changes is this re-roll v7:
> * Add get_comment_line_char subroutine to perl/Git.pm and use it.
> * get_comment_line_char gets the value of core.commentchar configuration
> variable. It handles the 'auto' value taking '#' in this case as the
> comment line character.
> * When core.commentchar is not set to one single character, take '#' as the
> comment line character. This follows the system behaviour programmed in
> config.c::git_default_core_config.
I gave it a read and I think it is ready to move forward.
Thanks.
^ permalink raw reply
* Re: [PATCHv2 5/5] rm: add absorb a submodules git dir before deletion
From: Junio C Hamano @ 2016-12-14 18:55 UTC (permalink / raw)
To: Stefan Beller; +Cc: git, David.Turner, bmwill, sandals
In-Reply-To: <20161213205622.841-6-sbeller@google.com>
Stefan Beller <sbeller@google.com> writes:
> diff --git a/builtin/rm.c b/builtin/rm.c
> index fdd7183f61..8c9c535a88 100644
> --- a/builtin/rm.c
> +++ b/builtin/rm.c
> @@ -392,28 +392,14 @@ int cmd_rm(int argc, const char **argv, const char *prefix)
> for (i = 0; i < list.nr; i++) {
> const char *path = list.entry[i].name;
> if (list.entry[i].is_submodule) {
> - if (is_empty_dir(path)) {
> - if (!rmdir(path)) {
> - removed = 1;
> - if (!remove_path_from_gitmodules(path))
> - gitmodules_modified = 1;
> - continue;
> - }
> - } else {
> - strbuf_reset(&buf);
> - strbuf_addstr(&buf, path);
> - if (!remove_dir_recursively(&buf, 0)) {
> - removed = 1;
> - if (!remove_path_from_gitmodules(path))
> - gitmodules_modified = 1;
> - strbuf_release(&buf);
> - continue;
> - } else if (!file_exists(path))
> - /* Submodule was removed by user */
> - if (!remove_path_from_gitmodules(path))
> - gitmodules_modified = 1;
> - /* Fallthrough and let remove_path() fail. */
> - }
> + if (is_empty_dir(path) && rmdir(path))
> + die_errno("git rm: '%s'", path);
> + if (file_exists(path))
> + depopulate_submodule(path);
> + removed = 1;
> + if (!remove_path_from_gitmodules(path))
> + gitmodules_modified = 1;
> + continue;
The updated code structure is somewhat nicer for understanding the
flow than the original, but it can further be improved.
A step "If empty directory, we try to rmdir and we check its return
status and die ourselves if we couldn't remove it" reads very
sensible, but coming immediately after that, "if it still exists,
call depop()" looks a bit strange. I would have expected a similar
construct, i.e.
if (directory_exists(path) && depop_submodlue(path))
die("git rm: '%s' submodule still populated", path);
there. Also,
if (is_empty_dir(path)) {
if (rmdir(path))
die_errno(...);
} else if (is_nonempty_dir(path)) {
if (depop_subm(path))
die(...);
}
would have clarified the structure even further.
Yes, I know that you made depop_subm() to die on error, and the
above is questioning if that is a sensible design choice.
^ permalink raw reply
* Re: [PATCHv2 4/5] ok_to_remove_submodule: absorb the submodule git dir
From: Stefan Beller @ 2016-12-14 18:52 UTC (permalink / raw)
To: Junio C Hamano
Cc: git@vger.kernel.org, David Turner, Brandon Williams,
brian m. carlson
In-Reply-To: <xmqqbmwexs9x.fsf@gitster.mtv.corp.google.com>
On Wed, Dec 14, 2016 at 10:46 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> It is a major reason to say no, when deciding if a submodule can be
>> deleted, if the git directory of the submodule being contained in the
>> submodule's working directory.
>>
>> Migrate the git directory into the superproject instead of failing,
>> and proceed with the other checks.
>>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> ---
>> submodule.c | 15 +++++++++++++--
>> 1 file changed, 13 insertions(+), 2 deletions(-)
>>
>> diff --git a/submodule.c b/submodule.c
>> index 2d13744b06..e42efa2337 100644
>> --- a/submodule.c
>> +++ b/submodule.c
>> @@ -1026,11 +1026,22 @@ int ok_to_remove_submodule(const char *path, unsigned flags)
>> struct strbuf buf = STRBUF_INIT;
>> int ok_to_remove = 1;
>>
>> + /* Is it there? */
>> if (!file_exists(path) || is_empty_dir(path))
>> return 1;
>>
>> - if (!submodule_uses_gitfile(path))
>> - return 0;
>> + /* Does it have a .git directory? */
>> + if (!submodule_uses_gitfile(path)) {
>> + absorb_git_dir_into_superproject("", path,
>> + ABSORB_GITDIR_RECURSE_SUBMODULES);
>> +
>> + /*
>> + * We should be using a gitfile by now. Let's double
>> + * check as losing the git dir would be fatal.
>> + */
>> + if (!submodule_uses_gitfile(path))
>> + return 0;
>> + }
>
> It feels a bit funny for a function that answers yes/no question to
> actually have a side effect, but probably it is OK. It feels dirty,
> but I dunno.
Another approach would be to return more than yes/no but the reason
why it is a no. And then the caller would call absorb_git_dir_into_superproject.
> A brief reading of the above makes us wonder what should happen if
> the absorb_git_dir_into_superproject() call fails, but a separate
> "submodule_uses_gitfile()" is needed to see if it failed? Perhaps
> the function needs to tell the caller if it succeeded?
I don't know about the double check as I think we should really be sure before
deleting that repo, but absorb_git_dir_into_superproject would fail/die
if it would not actually absorb it, so maybe it is too much caution.
So I'd omit the double check in a reroll.
>
>>
>> argv_array_pushl(&cp.args, "status", "--porcelain",
>> "--ignore-submodules=none", NULL);
^ permalink raw reply
* Re: [PATCHv2 4/5] ok_to_remove_submodule: absorb the submodule git dir
From: Junio C Hamano @ 2016-12-14 18:46 UTC (permalink / raw)
To: Stefan Beller; +Cc: git, David.Turner, bmwill, sandals
In-Reply-To: <20161213205622.841-5-sbeller@google.com>
Stefan Beller <sbeller@google.com> writes:
> It is a major reason to say no, when deciding if a submodule can be
> deleted, if the git directory of the submodule being contained in the
> submodule's working directory.
>
> Migrate the git directory into the superproject instead of failing,
> and proceed with the other checks.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
> submodule.c | 15 +++++++++++++--
> 1 file changed, 13 insertions(+), 2 deletions(-)
>
> diff --git a/submodule.c b/submodule.c
> index 2d13744b06..e42efa2337 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -1026,11 +1026,22 @@ int ok_to_remove_submodule(const char *path, unsigned flags)
> struct strbuf buf = STRBUF_INIT;
> int ok_to_remove = 1;
>
> + /* Is it there? */
> if (!file_exists(path) || is_empty_dir(path))
> return 1;
>
> - if (!submodule_uses_gitfile(path))
> - return 0;
> + /* Does it have a .git directory? */
> + if (!submodule_uses_gitfile(path)) {
> + absorb_git_dir_into_superproject("", path,
> + ABSORB_GITDIR_RECURSE_SUBMODULES);
> +
> + /*
> + * We should be using a gitfile by now. Let's double
> + * check as losing the git dir would be fatal.
> + */
> + if (!submodule_uses_gitfile(path))
> + return 0;
> + }
It feels a bit funny for a function that answers yes/no question to
actually have a side effect, but probably it is OK. It feels dirty,
but I dunno.
A brief reading of the above makes us wonder what should happen if
the absorb_git_dir_into_superproject() call fails, but a separate
"submodule_uses_gitfile()" is needed to see if it failed? Perhaps
the function needs to tell the caller if it succeeded?
>
> argv_array_pushl(&cp.args, "status", "--porcelain",
> "--ignore-submodules=none", NULL);
^ permalink raw reply
* Re: [PATCHv2 2/5] submodule: modernize ok_to_remove_submodule to use argv_array
From: Junio C Hamano @ 2016-12-14 18:39 UTC (permalink / raw)
To: Stefan Beller; +Cc: git, David.Turner, bmwill, sandals
In-Reply-To: <20161213205622.841-3-sbeller@google.com>
Stefan Beller <sbeller@google.com> writes:
> Instead of constructing the NULL terminated array ourselves, we
> should make use of the argv_array infrastructure.
>
> While at it, adapt the error messages to reflect the actual invocation.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
> submodule.c | 14 ++++----------
> 1 file changed, 4 insertions(+), 10 deletions(-)
Looks good.
>
> diff --git a/submodule.c b/submodule.c
> index 45ccfb7ab4..9f0b544ebe 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -1023,13 +1023,6 @@ int ok_to_remove_submodule(const char *path)
> {
> ssize_t len;
> struct child_process cp = CHILD_PROCESS_INIT;
> - const char *argv[] = {
> - "status",
> - "--porcelain",
> - "-u",
> - "--ignore-submodules=none",
> - NULL,
> - };
> struct strbuf buf = STRBUF_INIT;
> int ok_to_remove = 1;
>
> @@ -1039,14 +1032,15 @@ int ok_to_remove_submodule(const char *path)
> if (!submodule_uses_gitfile(path))
> return 0;
>
> - cp.argv = argv;
> + argv_array_pushl(&cp.args, "status", "--porcelain", "-u",
> + "--ignore-submodules=none", NULL);
> prepare_submodule_repo_env(&cp.env_array);
> cp.git_cmd = 1;
> cp.no_stdin = 1;
> cp.out = -1;
> cp.dir = path;
> if (start_command(&cp))
> - die("Could not run 'git status --porcelain -uall --ignore-submodules=none' in submodule %s", path);
> + die(_("could not run 'git status --porcelain -u --ignore-submodules=none' in submodule %s"), path);
>
> len = strbuf_read(&buf, cp.out, 1024);
> if (len > 2)
> @@ -1054,7 +1048,7 @@ int ok_to_remove_submodule(const char *path)
> close(cp.out);
>
> if (finish_command(&cp))
> - die("'git status --porcelain -uall --ignore-submodules=none' failed in submodule %s", path);
> + die(_("'git status --porcelain -u --ignore-submodules=none' failed in submodule %s"), path);
>
> strbuf_release(&buf);
> return ok_to_remove;
^ permalink raw reply
* Re: [PATCHv3 1/3] merge: Add '--continue' option as a synonym for 'git commit'
From: Junio C Hamano @ 2016-12-14 18:04 UTC (permalink / raw)
To: Chris Packham; +Cc: git, mah, peff, jacob.keller
In-Reply-To: <20161214083757.26412-1-judge.packham@gmail.com>
The last one 3/3 is a nice touch that makes sure that we do not
forget what we discovered during the discussion. Very much
appreciated.
Will queue. Thanks.
^ permalink raw reply
* Re: [PATCH v9 4/5] http: create function to get curl allowed protocols
From: Brandon Williams @ 2016-12-14 18:00 UTC (permalink / raw)
To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder
In-Reply-To: <20161214160330.iqvwxshsgk4n2gm7@sigill.intra.peff.net>
On 12/14, Jeff King wrote:
> On Tue, Dec 13, 2016 at 05:40:36PM -0800, Brandon Williams wrote:
>
> > Move the creation of an allowed protocols whitelist to a helper
> > function.
>
> This is "what" but not "why". You can figure it out if you see the next
> patch, but it's often nice to make a brief mention, like:
>
> This will be useful when we need to compute the set of allowed
> protocols differently for normal and redirect cases.
Commit message writing is hard (at least for me :). I'll update the
message to indicate the why.
--
Brandon Williams
^ permalink raw reply
* Re: [PATCH] parse-options: print "fatal:" before usage_msg_opt()
From: Junio C Hamano @ 2016-12-14 17:57 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20161214151009.4wdzjb44f6aki5ug@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
> Programs may use usage_msg_opt() to print a brief message
> followed by the program usage, and then exit. The message
> isn't prefixed at all, though, so it doesn't match our usual
> error output and is easy to overlook:
>
> $ git clone 1 2 3
> Too many arguments.
>
> usage: git clone [<options>] [--] <repo> [<dir>]
>
> -v, --verbose be more verbose
> -q, --quiet be more quiet
> --progress force progress reporting
> -n, --no-checkout don't create a checkout
> --bare create a bare repository
> [...and so on for another 31 lines...]
>
> It looks especially bad when the message starts with an
> option, like:
>
> $ git replace -e
> -e needs exactly one argument
>
> usage: git replace [-f] <object> <replacement>
> or: git replace [-f] --edit <object>
> [...etc...]
>
> Let's put our usual "fatal:" prefix in front of it.
I briefly wondered if any caller uses this in a non-fatal situation,
but usage_with_options() always dies, so this looks like the right
thing to do. Thanks.
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
> Some of the message in git-clone could stand to be rewritten to match
> our usual style, too (no capitals, no trailing period), but that's
> obviously out of scope for this patch. I don't think this change makes
> them look any worse.
>
> parse-options.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/parse-options.c b/parse-options.c
> index 312a85dbd..4fbe924a5 100644
> --- a/parse-options.c
> +++ b/parse-options.c
> @@ -661,7 +661,7 @@ void NORETURN usage_msg_opt(const char *msg,
> const char * const *usagestr,
> const struct option *options)
> {
> - fprintf(stderr, "%s\n\n", msg);
> + fprintf(stderr, "fatal: %s\n\n", msg);
> usage_with_options(usagestr, options);
> }
^ permalink raw reply
* Re: [PATCH v9 3/5] http: always warn if libcurl version is too old
From: Brandon Williams @ 2016-12-14 17:56 UTC (permalink / raw)
To: Jeff King; +Cc: git, sbeller, bburky, gitster, jrnieder
In-Reply-To: <20161214160123.a6a7fve5qz5xgg7n@sigill.intra.peff.net>
On 12/14, Jeff King wrote:
> On Tue, Dec 13, 2016 at 05:40:35PM -0800, Brandon Williams wrote:
>
> > diff --git a/transport.c b/transport.c
> > index e1ba78b..fbd799d 100644
> > --- a/transport.c
> > +++ b/transport.c
> > @@ -700,11 +700,6 @@ void transport_check_allowed(const char *type)
> > die("transport '%s' not allowed", type);
> > }
> >
> > -int transport_restrict_protocols(void)
> > -{
> > - return !!protocol_whitelist();
> > -}
> > -
>
> This function was subtly broken as of patch 2 of the series. It's
> probably not a big deal in the long run, but should the series be
> re-ordered to put this one first?
>
> I think the commit message would need adjusted, but it probably should
> mention the reasons this is a good idea even _without_ the new config
> system. Namely that even when there's no protocol whitelist, newer
> versions of curl have all of the other non-http protocols disabled.
>
> I wonder if anybody is actually using a version of curl old enough to
> trigger this. If so, they're going to get the warning every time they
> fetch via http. We might need to stick it behind an "advice.*" config
> option, though I'm inclined to leave it as-is and see if anybody
> actually complains.
>
> -Peff
Yeah you're right, transport_restrict_protocols() is definitely broken
after patch 2. Since I'm probably going to need to do a reroll based on
some of your comments in the other patches in the series we might as
well reorder patch 2 and 3 so this isn't broken between patches.
--
Brandon Williams
^ permalink raw reply
* Re: [PATCH v2 4/6] update-unicode.sh: automatically download newer definition files
From: Junio C Hamano @ 2016-12-14 17:50 UTC (permalink / raw)
To: Beat Bolli; +Cc: Git List
In-Reply-To: <b137249e-728a-5d3c-4993-5ed5a1593737@drbeat.li>
Beat Bolli <dev+git@drbeat.li> writes:
> On 14.12.16 00:31, Beat Bolli wrote:
>
>> [PATCH v2 4/6] update-unicode.sh: automatically download newer definition files
>
> Dang! And again I'm not capable of putting an underline instead of the
> dash...
>
> Junio, would you please reword the subject to
>
> Re: [PATCH v2 4/6] update_unicode.sh: automatically download newer
> definition files
Will do.
This is an indication that the script is probably named against
people's expectation. We may want to rename it after the dust
settles.
Thanks.
^ permalink raw reply
* Re: [PATCH v2 4/6] update-unicode.sh: automatically download newer definition files
From: Beat Bolli @ 2016-12-14 17:40 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git List
In-Reply-To: <1481671904-1143-5-git-send-email-dev+git@drbeat.li>
On 14.12.16 00:31, Beat Bolli wrote:
> [PATCH v2 4/6] update-unicode.sh: automatically download newer definition files
Dang! And again I'm not capable of putting an underline instead of the
dash...
Junio, would you please reword the subject to
Re: [PATCH v2 4/6] update_unicode.sh: automatically download newer
definition files
Thanks,
Beat
> we should also download them if a newer version exists on the Unicode
> consortium's servers. Option -N of wget does this nicely for us.
>
> Reviewed-by: Torsten Bögershausen <tboegi@web.de>
> Signed-off-by: Beat Bolli <dev+git@drbeat.li>
> ---
> contrib/update-unicode/update_unicode.sh | 8 ++------
> 1 file changed, 2 insertions(+), 6 deletions(-)
>
> diff --git a/contrib/update-unicode/update_unicode.sh b/contrib/update-unicode/update_unicode.sh
> index 9f1bf31..56871a1 100755
> --- a/contrib/update-unicode/update_unicode.sh
> +++ b/contrib/update-unicode/update_unicode.sh
> @@ -8,12 +8,8 @@
> cd "$(dirname "$0")"
> UNICODEWIDTH_H=$(git rev-parse --show-toplevel)/unicode_width.h
>
> -if ! test -f UnicodeData.txt; then
> - wget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt
> -fi &&
> -if ! test -f EastAsianWidth.txt; then
> - wget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt
> -fi &&
> +wget -N http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt \
> + http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt &&
> if ! test -d uniset; then
> git clone https://github.com/depp/uniset.git &&
> ( cd uniset && git checkout 4b186196dd )
>
^ permalink raw reply
* Re: [PATCHv2 0/7] Fix and generalize version sort reordering
From: Junio C Hamano @ 2016-12-14 17:36 UTC (permalink / raw)
To: Jeff King
Cc: SZEDER Gábor, Nguyễn Thái Ngọc Duy,
Leho Kraav, git
In-Reply-To: <20161214170852.bzh5pyl4bov6rwbt@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
> On Thu, Dec 08, 2016 at 03:23:54PM +0100, SZEDER Gábor wrote:
>
>> > With my patches it looks like this:
>> >
>> > $ git -c versionsort.prereleasesuffix=-pre \
>> > -c versionsort.prereleasesuffix=-prerelease \
>> > tag -l --sort=version:refname
>> > v1.0.0-prerelease1
>> > v1.0.0-pre1
>> > v1.0.0-pre2
>> > v1.0.0
>>
>> And when there happen to be more than one matching suffixes starting
>> at the same earliest position, then we should pick the longest of
>> them. The new patch 6/7 implements this behavior.
>
> The whole approach taken by the suffix code (before your patches) leaves
> me with the nagging feeling that the comparison is not always going to
> be transitive (i.e., that "a < b && b < c" does not always imply "a <
> c"), which is going to cause nonsensical sorting results.
>
> And that may be part of the issue your 6/7 fixes.
>
> I spent some time playing with this the other day, though, and couldn't
> come up with a specific example that fails the condition above.
>
> It just seems like the whole thing would conceptually easier if we
> pre-parsed the versions into a sequence of elements, then the comparison
> between any two elements would just walk that sequence. The benefit
> there is that you can implement whatever rules you like for the parsing
> (like "prefer longer suffixes to shorter"), but you know the comparison
> will always be consistent.
>
> It would also be more efficient, I think (it seems like the sort is
> O(nr_tags * lg(nr_tags) * nr_suffixes) due to parsing suffixes in the
> comparator). Though that probably doesn't matter much in practice.
>
> I dunno. I think maybe your 6/7 has converged on an equivalent behavior.
> And I am certainly not volunteering to re-write it, so if what you have
> works, I'm not opposed to it.
I also had worries about transitiveness but couldn't break it after
trying for some time. I find your pre-parsing suggestion a great
one, not from the point of view of performance, but because I would
imagine that the resulting logic would become a lot easier to
understand.
^ permalink raw reply
* Re: [PATCH] fix pushing to //server/share/dir paths on Windows
From: Jeff King @ 2016-12-14 17:30 UTC (permalink / raw)
To: Johannes Sixt; +Cc: Johannes Schindelin, Git Mailing List
In-Reply-To: <2ff2613c-47da-a780-5d38-93e16cb16328@kdbg.org>
On Tue, Dec 13, 2016 at 10:32:01PM +0100, Johannes Sixt wrote:
> normalize_path_copy() is not prepared to keep the double-slash of a
> //server/share/dir kind of path, but treats it like a regular POSIX
> style path and transforms it to /server/share/dir.
>
> The bug manifests when 'git push //server/share/dir master' is run,
> because tmp_objdir_add_as_alternate() uses the path in normalized
> form when it registers the quarantine object database via
> link_alt_odb_entries(). Needless to say that the directory cannot be
> accessed using the wrongly normalized path.
Thanks for digging this up! I had a feeling that the problem was going
to be in the underlying path code, but I didn't want to just pass the
buck without evidence. :)
> - if (is_dir_sep(*src)) {
> + /*
> + * Handle initial part of absolute path: "/", "C:/", "\\server\share/".
> + */
> + offset = offset_1st_component(src);
> + if (offset) {
> + /* Convert the trailing separator to '/' on Windows. */
> + memcpy(dst, src, offset - 1);
> + dst += offset - 1;
> *dst++ = '/';
Hmm. So this is the "change-of-behavior" bit. Would it be reasonable to
write:
/* Copy initial part of absolute path, converting separators on Windows */
const char *end = src + offset_1st_component(src);
while (src < end) {
char c = *src++;
if (c == '\\')
c = '/';
*dst++ = c;
}
? I'm not sure if it's wrong to convert backslashes in that first
component or not (but certainly we were before). I don't think we'd need
is_dir_sep() in that "if()", because we can leave slashes as-is. But
maybe it would make the code easier to read.
-Peff
^ permalink raw reply
* Re: [PATCHv2 0/7] Fix and generalize version sort reordering
From: Jeff King @ 2016-12-14 17:08 UTC (permalink / raw)
To: SZEDER Gábor
Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy, Leho Kraav,
git
In-Reply-To: <20161208142401.1329-1-szeder.dev@gmail.com>
On Thu, Dec 08, 2016 at 03:23:54PM +0100, SZEDER Gábor wrote:
> > With my patches it looks like this:
> >
> > $ git -c versionsort.prereleasesuffix=-pre \
> > -c versionsort.prereleasesuffix=-prerelease \
> > tag -l --sort=version:refname
> > v1.0.0-prerelease1
> > v1.0.0-pre1
> > v1.0.0-pre2
> > v1.0.0
>
> And when there happen to be more than one matching suffixes starting
> at the same earliest position, then we should pick the longest of
> them. The new patch 6/7 implements this behavior.
The whole approach taken by the suffix code (before your patches) leaves
me with the nagging feeling that the comparison is not always going to
be transitive (i.e., that "a < b && b < c" does not always imply "a <
c"), which is going to cause nonsensical sorting results.
And that may be part of the issue your 6/7 fixes.
I spent some time playing with this the other day, though, and couldn't
come up with a specific example that fails the condition above.
It just seems like the whole thing would conceptually easier if we
pre-parsed the versions into a sequence of elements, then the comparison
between any two elements would just walk that sequence. The benefit
there is that you can implement whatever rules you like for the parsing
(like "prefer longer suffixes to shorter"), but you know the comparison
will always be consistent.
It would also be more efficient, I think (it seems like the sort is
O(nr_tags * lg(nr_tags) * nr_suffixes) due to parsing suffixes in the
comparator). Though that probably doesn't matter much in practice.
I dunno. I think maybe your 6/7 has converged on an equivalent behavior.
And I am certainly not volunteering to re-write it, so if what you have
works, I'm not opposed to it.
-Peff
^ permalink raw reply
* Re: [PATCHv3 1/3] merge: Add '--continue' option as a synonym for 'git commit'
From: Junio C Hamano @ 2016-12-14 17:01 UTC (permalink / raw)
To: Jeff King; +Cc: Chris Packham, git, mah, jacob.keller
In-Reply-To: <20161214152039.swtll7xrmcdwz7bc@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
> So not too bad (and you could probably refactor it to avoid some of the
> duplication). Though it does get some obscure cases wrong, like:
>
> git merge --continue --verbose --quiet
>
> I dunno. Maybe I am leading you down a rabbit hole, and we should just
> live with silently ignoring useless options.
I think you need to handle this in parse-options API if you really
wanted to do this correctly.
<xmqq60mn671x.fsf@gitster.mtv.corp.google.com> may serve as a
reasonable outline for building one.
^ permalink raw reply
* Re: [PATCH v9 5/5] transport: add from_user parameter to is_transport_allowed
From: Jeff King @ 2016-12-14 16:40 UTC (permalink / raw)
To: Brandon Williams; +Cc: git, sbeller, bburky, gitster, jrnieder
In-Reply-To: <1481679637-133137-6-git-send-email-bmwill@google.com>
On Tue, Dec 13, 2016 at 05:40:37PM -0800, Brandon Williams wrote:
> Add the from_user parameter to the 'is_transport_allowed' function.
> This allows callers to query if a transport protocol is allowed, given
> that the caller knows that the protocol is coming from the user (1) or
> not from the user (0) such as redirects in libcurl. If unknown a -1
> should be provided which falls back to reading `GIT_PROTOCOL_FROM_USER`
> to determine if the protocol came from the user.
I think your commit message is upside-down with respect to the purpose
of the patch. The end goal we want is for http to distinguish between
protocol restrictions for redirects versus initial requests. The rest is
an implementation detail. It's definitely still worth discussing that
implementation detail (though I think your in-code comments may be
sufficient), but I don't see the rationale discussed here at all.
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
> http.c | 14 +++++++-------
> transport.c | 8 +++++---
> transport.h | 13 ++++++++++---
> 3 files changed, 22 insertions(+), 13 deletions(-)
I'm trying to think of a way to test this. I guess the case we are
covering here is when a server redirects, but the protocol is only
allowed from the user. So:
diff --git a/t/t5812-proto-disable-http.sh b/t/t5812-proto-disable-http.sh
index 044cc152f..d911afd24 100755
--- a/t/t5812-proto-disable-http.sh
+++ b/t/t5812-proto-disable-http.sh
@@ -30,5 +30,12 @@ test_expect_success 'curl limits redirects' '
test_must_fail git clone "$HTTPD_URL/loop-redir/smart/repo.git"
'
+test_expect_success 'http can be limited to from-user' '
+ git -c protocol.http.allow=user \
+ clone "$HTTPD_URL/smart/repo.git" plain.git &&
+ test_must_fail git -c protocol.http.allow=user \
+ clone "$HTTPD_URL/smart-redir-perm/repo.git" redir.git
+'
+
stop_httpd
test_done
It's an oddball configuration, and you'd probably just set
http.followRedirects=false in practice, but it does correctly check this
case.
> @@ -588,9 +588,9 @@ static CURL *get_curl_handle(void)
> #endif
> #if LIBCURL_VERSION_NUM >= 0x071304
> curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS,
> - get_curl_allowed_protocols());
> + get_curl_allowed_protocols(0));
> curl_easy_setopt(result, CURLOPT_PROTOCOLS,
> - get_curl_allowed_protocols());
> + get_curl_allowed_protocols(-1));
This covers internal redirects done by libcurl, but not the dumb-walker
http-alternates nonsense. We have to feed the URL from http-alternates
back to curl ourselves, so it uses CURLOPT_PROTOCOLS even though it
should count as "not from the user".
To fix that, I think we'd need something like:
- get_curl_handle() stops setting these options, as it is done only
once when the curl handle is initialized. Instead, the protocol
restrictions should go into get_active_slot(), which is called for
each request. The values set would remain the same, and be the
baseline.
- the http-walker.c code would need to know when it's requesting from
the base URL, and when it's an alternate. I think this would depend
on the position of the "alt" in in the linked list it keeps.
- when requesting from an alternate, http-walker would set
CURLOPT_PROTOCOLS with get_curl_allowed_protocols(0)
I have to admit that it sounds like a fair bit of work for a pretty
obscure case. You'd have to:
1. Turn http.allowRedirects to "true", to allow redirects even for
non-initial contact.
2. Turn one of protocol.{http,https,ftp,ftps}.allow to "user" to
restrict it from being used in a redirect.
I'm tempted to punt on it and just do:
if (http_follow_config == HTTP_FOLLOW_ALWAYS &&
get_curl_allowed_protocols(0) != get_curl_allowed_protocols(-1))
die("user-only protocol restrictions not implemented for http-alternates");
which errs on the safe side. We could even shove that down into the case
where we actually see some alternates, like:
diff --git a/http-walker.c b/http-walker.c
index c2f81cd6a..5bcc850b1 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -160,6 +160,12 @@ static void prefetch(struct walker *walker, unsigned char *sha1)
#endif
}
+static void check_alternates_protocol_restrictions(void)
+{
+ if (get_curl_allowed_protocols(0) != get_curl_allowed_protocol(-1))
+ die("user-only protocol restrictions not implemented for http alternates");
+}
+
static void process_alternates_response(void *callback_data)
{
struct alternates_request *alt_req =
@@ -272,6 +278,7 @@ static void process_alternates_response(void *callback_data)
/* skip "objects\n" at end */
if (okay) {
struct strbuf target = STRBUF_INIT;
+ check_alternates_protocol_restrictions();
strbuf_add(&target, base, serverlen);
strbuf_add(&target, data + i, posn - i - 7);
warning("adding alternate object store: %s",
I find it unlikely that anybody would ever care, but at least we'd do
the safe thing. I dunno. Maybe I am just being lazy.
-Peff
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox