* Re: [PATCH] rebase: mark --update-refs as requiring the merge backend
From: Elijah Newren @ 2023-01-20 16:47 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Derrick Stolee, Elijah Newren via GitGitGadget, git
In-Reply-To: <xmqqr0vpxm3d.fsf@gitster.g>
On Fri, Jan 20, 2023 at 7:27 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Derrick Stolee <derrickstolee@github.com> writes:
>
> >> + if (options.update_refs)
> >> + imply_merge(&options, "--update-refs");
> >> +
> >
> > This solution is very elegant. The only downside is the lack of warning
> > if --update-refs was implied by rebase.updateRefs=true, but I'm happy to
> > delay implementing that warning in favor of your complete solution here.
>
> If features A and B are incompatible and both can be specified from
> both command line and configuration, ideally I would expect the
> system to operate in one of two ways.
I agree that one of the two ways you highlight below would be ideal.
Should this series be held up on extending the checks to implement
this ideal, or are you just commenting for whoever later circles back
to implement this?
> I haven't thought it through
> to decide which one I prefer between the two.
>
> * Take "command line trumps configuration" one step further, so
> that A that is configured but not asked for from the command
> line is defeated by B that is asked for from the command line.
>
> This way, only when A and B are both requested via the
> configuration, of via the command line, we'd fail the operation
> by saying A and B are incompatible. Otherwise, the one that is
> configured but overridden is turned off (either silently or with
> a warning).
>
> * Declare "command line trumps configuration" is only among the
> same feature. Regardless of how features A and B that are
> incompatible are requested, the command will error out, citing
> incompatibility. It would be very nice if the warning mentioned
> where the requests for features A and B came from (e.g. "You
> asked for -B from the command line, but you have A configured,
> and both cannot be active at the same time---disable A from the
> command line, or do not ask for B")
>
> When A is configured and B is requested from the command line,
> the command will error out, and the user must defeat A from the
> command line before the user can use B, e.g. "git cmd --no-A -B".
>
> A knee-jerk reaction to the situation is that the latter feels
> somewhat safer than the former, but when I imagine the actual end
> user who saw the error message, especially the suggested solution
> "disable A from the command line or do not ask for B from the
> command line", may say "well, I asked for B for this invocation
> explicitly with -B from the command line, and you(Git) should be
> able to make it imply --no-A", which amounts to the same thing as
> the former choice.
If it is clear to the user that A and B preclude each other, then I
agree with this sentiment that the former choice (silently ignoring
the config) would avoid a minor frustration for some users and thus
would be better. But I don't think that's applicable here. There is
no reason that --whitespace=fix shouldn't be available from the merge
backend other than that we haven't implemented it yet, and it's likely
feasible to implement --update-refs for the apply backend with enough
effort if we thought it was worth it. So, if a user sets
rebase.updateRefs=true in their config because they always want
included branches updated, but one time they run `git rebase
--whitespace=fix`, they will likely have a negative experience like
the one that inspired this patch. Perhaps we're forced to choose
between possible frustration by different end users, but if so, I
think trying to debug and figure out "Wait, I switched to this branch
and started tweaking it but it appears to not have some relevant
changes I'm sure I made to it yesterday. What happened?" is a much
worse frustration than "I have to manually specify --no-A in this
special case". So, when it's not at all obvious that A and B preclude
each other, I think we're better off giving the error.
^ permalink raw reply
* Re: [PATCH v2] fsm-listen-darwin: combine bit operations
From: Junio C Hamano @ 2023-01-20 17:52 UTC (permalink / raw)
To: Jeff Hostetler; +Cc: Rose via GitGitGadget, git, Seija Kijin
In-Reply-To: <021ab1ab-b90a-5a24-23c4-44e46d87c476@jeffhostetler.com>
Jeff Hostetler <git@jeffhostetler.com> writes:
>> static int ef_is_dropped(const FSEventStreamEventFlags ef)
>> {
>> - return (ef & kFSEventStreamEventFlagMustScanSubDirs ||
>> - ef & kFSEventStreamEventFlagKernelDropped ||
>> - ef & kFSEventStreamEventFlagUserDropped);
>> + return (ef & (kFSEventStreamEventFlagMustScanSubDirs |
>> + kFSEventStreamEventFlagKernelDropped |
>> + kFSEventStreamEventFlagUserDropped));
>> }
>
> Technically, the returned value is slightly different, but
> the only caller is just checking for non-zero, so it doesn't
> matter.
>
> So this is fine.
But is it worth the code churn and reviewer bandwidth? Don't we
have better things to spend our time on?
I would not be surprised if a smart enough compiler used the same
transformartion as this patch does manually as an optimization.
Then it matters more which one of the two is more readable by our
developers. And the original matches how we humans would think, I
would imagine. ef might have MustScanSubdirs bit, KernelDropped
bit, or UserDropped bit and in these cases we want to say that ef is
dropped. Arguably, the original is more readble, and it would be a
good change to adopt if there is an upside, like the updated code
resulting in markedly more efficient binary.
So, this might be technically fine, but I am not enthused to see
these kind of code churning patches with dubious upside. An
optimization patch should be able to demonstrate its benefit with a
solid benchmark, or at least a clear difference in generated code.
In fact.
Compiler explorer godbolt.org tells me that gcc 12 with -O2 compiles
the following two functions into identical assembly. The !! prefix
used in the second example is different from the postimage of what
Seija posted, but this being a file-scope static function, I would
expect the compiler to notice that the actual value would not matter
to the callers, only the truth value, does.
* Input *
int one(unsigned int num) {
return ((num & 01) ||
(num & 02) || (num & 04));
}
int two(unsigned int num) {
return !!((num) & (01|02|04));
}
* Assembly *
one(unsigned int):
xor eax, eax
and edi, 7
setne al
ret
two(unsigned int):
xor eax, eax
and edi, 7
setne al
ret
^ permalink raw reply
* [PATCH] Makefile: suppress annotated leaks with certain ASan options
From: Taylor Blau @ 2023-01-20 18:46 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Jeff King, Johannes Schindelin, Junio C Hamano,
Victoria Dye
When building with `SANITIZE=leak`, we define `SUPPRESS_ANNOTATED_LEAKS`
in order to make the `UNLEAK()` macro work (without the aforementioned
define, `UNLEAK()` is a noop). This is from `UNLEAK()`'s introduction in
0e5bba53af (add UNLEAK annotation for reducing leak false positives,
2017-09-08), where `UNLEAK()` is a noop for performance reasons unless
we are using the leak sanitizer.
However, it is possible to use the leak sanitizer without
`SANITIZE=leak`. This happens when building with `SANITIZE=address` and
enabling the leak sanitizer via the `ASAN_OPTIONS` variable (by
including the string "detect_leaks=1").
This renders `UNLEAK()` useless when doing `SANITIZE=address` builds
which also use the leak checker.
Update our Makefile to pretend as if `SANITIZE=leak` was given when
`SANITIZE=address` is given and the leak checker is enabled via
`ASAN_OPTIONS`.
Playing around with all five options (two spelling "enabled", two
spelling "disabled", and the empty set of options) yields the correct
behavior:
for opt in '' detect_leaks=1 detect_leaks=true detect_leaks=0 detect_leaks=false
do
echo "==> ${opt:-(nothing)}"
make -B builtin/add.o V=1 SANITIZE=address ASAN_OPTIONS="$opt" 2>&1 |
grep -o -- '-DSUPPRESS_ANNOTATED_LEAKS'
done
gives us:
==> (nothing)
-DSUPPRESS_ANNOTATED_LEAKS
==> detect_leaks=1
-DSUPPRESS_ANNOTATED_LEAKS
==> detect_leaks=true
-DSUPPRESS_ANNOTATED_LEAKS
==> detect_leaks=0
==> detect_leaks=false
Making it possible to rely on `UNLEAK()` when implicitly using the leak
checker via SANITIZE=address builds.
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
I found this while playing around with GitHub's ASan-enabled CI builds
for our internal fork following a merge with v2.38.3.
The check-chainlint recipe in t/Makefile started using "git diff" via
d00113ec34 (t/Makefile: apply chainlint.pl to existing self-tests,
2022-09-01), which triggered a leak in some of GitHub's custom code. I
was surprised when marking the variable with UNLEAK() didn't do the
trick, and ended up down this rabbit hole ;-).
Makefile | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index db447d0738..b00bb8bd1e 100644
--- a/Makefile
+++ b/Makefile
@@ -1445,13 +1445,18 @@ ifneq ($(filter undefined,$(SANITIZERS)),)
BASIC_CFLAGS += -DSHA1DC_FORCE_ALIGNED_ACCESS
endif
ifneq ($(filter leak,$(SANITIZERS)),)
-BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS
-BASIC_CFLAGS += -O0
SANITIZE_LEAK = YesCompiledWithIt
endif
ifneq ($(filter address,$(SANITIZERS)),)
NO_REGEX = NeededForASAN
SANITIZE_ADDRESS = YesCompiledWithIt
+ifeq ($(filter $(patsubst detect_leaks=%,%,$(ASAN_OPTIONS)),0 false),)
+SANITIZE_LEAK = YesViaASanOptions
+endif
+endif
+ifneq ($(SANITIZE_LEAK),)
+BASIC_CFLAGS += -DSUPPRESS_ANNOTATED_LEAKS
+BASIC_CFLAGS += -O0
endif
endif
--
2.38.0.16.g393fd4c6db
^ permalink raw reply related
* Re: [PATCH] Makefile: suppress annotated leaks with certain ASan options
From: Junio C Hamano @ 2023-01-20 19:41 UTC (permalink / raw)
To: Taylor Blau
Cc: git, Derrick Stolee, Jeff King, Johannes Schindelin, Victoria Dye
In-Reply-To: <b1efe56ab5193d5505ccb9334f7d15e1795c27fb.1674240261.git.me@ttaylorr.com>
Taylor Blau <me@ttaylorr.com> writes:
> However, it is possible to use the leak sanitizer without
> `SANITIZE=leak`. This happens when building with `SANITIZE=address` and
> enabling the leak sanitizer via the `ASAN_OPTIONS` variable (by
> including the string "detect_leaks=1").
Yuck. I cannot tell if this falls into "don't do it then if it
hurts" or pretty common thing people do that is worth helping.
> Making it possible to rely on `UNLEAK()` when implicitly using the leak
> checker via SANITIZE=address builds.
But as long as you did all the work, sure, why not ;-).
> I found this while playing around with GitHub's ASan-enabled CI builds
> for our internal fork following a merge with v2.38.3.
>
> The check-chainlint recipe in t/Makefile started using "git diff" via
> d00113ec34 (t/Makefile: apply chainlint.pl to existing self-tests,
> 2022-09-01), which triggered a leak in some of GitHub's custom code. I
> was surprised when marking the variable with UNLEAK() didn't do the
> trick, and ended up down this rabbit hole ;-).
Thanks. Will queue.
^ permalink raw reply
* Re: [PATCH v2] fsm-listen-darwin: combine bit operations
From: Jeff Hostetler @ 2023-01-20 19:48 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Rose via GitGitGadget, git, Seija Kijin
In-Reply-To: <xmqqwn5hw0t5.fsf@gitster.g>
On 1/20/23 12:52 PM, Junio C Hamano wrote:
> Jeff Hostetler <git@jeffhostetler.com> writes:
>
>>> static int ef_is_dropped(const FSEventStreamEventFlags ef)
>>> {
>>> - return (ef & kFSEventStreamEventFlagMustScanSubDirs ||
>>> - ef & kFSEventStreamEventFlagKernelDropped ||
>>> - ef & kFSEventStreamEventFlagUserDropped);
>>> + return (ef & (kFSEventStreamEventFlagMustScanSubDirs |
>>> + kFSEventStreamEventFlagKernelDropped |
>>> + kFSEventStreamEventFlagUserDropped));
>>> }
>>
>> Technically, the returned value is slightly different, but
>> the only caller is just checking for non-zero, so it doesn't
>> matter.
>>
>> So this is fine.
>
> But is it worth the code churn and reviewer bandwidth? Don't we
> have better things to spend our time on?
>
> I would not be surprised if a smart enough compiler used the same
> transformartion as this patch does manually as an optimization.
>
> Then it matters more which one of the two is more readable by our
> developers. And the original matches how we humans would think, I
> would imagine. ef might have MustScanSubdirs bit, KernelDropped
> bit, or UserDropped bit and in these cases we want to say that ef is
> dropped. Arguably, the original is more readble, and it would be a
> good change to adopt if there is an upside, like the updated code
> resulting in markedly more efficient binary.
>
> So, this might be technically fine, but I am not enthused to see
> these kind of code churning patches with dubious upside. An
> optimization patch should be able to demonstrate its benefit with a
> solid benchmark, or at least a clear difference in generated code.
>
> In fact.
>
> Compiler explorer godbolt.org tells me that gcc 12 with -O2 compiles
> the following two functions into identical assembly. The !! prefix
> used in the second example is different from the postimage of what
> Seija posted, but this being a file-scope static function, I would
> expect the compiler to notice that the actual value would not matter
> to the callers, only the truth value, does.
>
>
> * Input *
> int one(unsigned int num) {
> return ((num & 01) ||
> (num & 02) || (num & 04));
> }
>
> int two(unsigned int num) {
> return !!((num) & (01|02|04));
> }
>
> * Assembly *
> one(unsigned int):
> xor eax, eax
> and edi, 7
> setne al
> ret
> two(unsigned int):
> xor eax, eax
> and edi, 7
> setne al
> ret
agreed. i didn't think the change was really worth the bother
and churn. personally, i prefer the conceptual clarity of the
code the way I wrote it.
and i was wondering if the compiler would generate the same
result, but didn't take the time (read: was too lazy) to actually
verify that.
all i was intending to say was that it wasn't a wrong change.
jeff
^ permalink raw reply
* Re: [PATCH] Makefile: suppress annotated leaks with certain ASan options
From: Jeff King @ 2023-01-20 20:15 UTC (permalink / raw)
To: Taylor Blau
Cc: git, Derrick Stolee, Johannes Schindelin, Junio C Hamano,
Victoria Dye
In-Reply-To: <b1efe56ab5193d5505ccb9334f7d15e1795c27fb.1674240261.git.me@ttaylorr.com>
On Fri, Jan 20, 2023 at 01:46:16PM -0500, Taylor Blau wrote:
> However, it is possible to use the leak sanitizer without
> `SANITIZE=leak`. This happens when building with `SANITIZE=address` and
> enabling the leak sanitizer via the `ASAN_OPTIONS` variable (by
> including the string "detect_leaks=1").
>
> This renders `UNLEAK()` useless when doing `SANITIZE=address` builds
> which also use the leak checker.
Yeah. I focused on LSan when adding the sanitize/unleak infrastructure
just because it was faster than a full ASan run, which made iterating on
fixes easier. I do think in the long run, once the test suite is leak
free, we may want to support leak-checking via ASan for the simple
reason that it can be done for "free" during the existing ASan
build/test, rather than requiring an extra LSan job.
I do think there's some complexity here, though.
One problem UNLEAK() is that compile-time switch, but whether ASan does
leak detection is a run-time choice. So you are stuck with either:
- you always turn on UNLEAK() for ASan builds, in which case test runs
using the default ASAN_OPTIONS we set do the extra work even though
they are not doing any leak detection. I doubt it's very measurable,
though (it's just shoving a few bytes onto a linked list),
especially compared to the overall slowness of ASan.
- you predicate the build-time choice on ASAN_OPTIONS. But this means
that:
make SANITIZE=address
cd t
ASAN_OPTIONS=detect_leaks=1 ./t0000-*.sh
will confusingly fail to use UNLEAK().
Your patch does the second one, but I think the first may be the
least-bad option.
The other issue I'd worry about is optimizations. Generally you want to
use -O2 with ASan, because it speeds things up (even more than just
regular -O2, I think, because it is optimizing the ASan instrumentation
code, too). I don't know offhand of any cases where it would find or not
find cases based on optimization level, though I could believe they
exist.
But for leak-checking, we've already seen real cases where using LSan
with higher optimization levels can lead to false positives (because the
optimizer drops a value that is still in scope but not used in a code
path that leads to exit()).
So it may be that we really do want to keep leak-checking to "-O0
-fsanitize=leak", and reserve "-O2 -fsanitize=address" for finding
address bugs.
> I found this while playing around with GitHub's ASan-enabled CI builds
> for our internal fork following a merge with v2.38.3.
>
> The check-chainlint recipe in t/Makefile started using "git diff" via
> d00113ec34 (t/Makefile: apply chainlint.pl to existing self-tests,
> 2022-09-01), which triggered a leak in some of GitHub's custom code. I
> was surprised when marking the variable with UNLEAK() didn't do the
> trick, and ended up down this rabbit hole ;-).
Yes, but I'd ask why your ASan builds were checking for leaks in the
first place. There are presumably tons of leaks they'd detect, since the
test suite is far from leak-free.
-Peff
^ permalink raw reply
* Re: [PATCH] Makefile: suppress annotated leaks with certain ASan options
From: Junio C Hamano @ 2023-01-20 20:46 UTC (permalink / raw)
To: Jeff King
Cc: Taylor Blau, git, Derrick Stolee, Johannes Schindelin,
Victoria Dye
In-Reply-To: <Y8r2Tn75g52YTIij@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
> I do think there's some complexity here, though.
>
> One problem UNLEAK() is that compile-time switch, but whether ASan does
> leak detection is a run-time choice. So you are stuck with either:
>
> - you always turn on UNLEAK() for ASan builds, in which case test runs
> using the default ASAN_OPTIONS we set do the extra work even though
> they are not doing any leak detection. I doubt it's very measurable,
> though (it's just shoving a few bytes onto a linked list),
> especially compared to the overall slowness of ASan.
>
> - you predicate the build-time choice on ASAN_OPTIONS. But this means
> that:
>
> make SANITIZE=address
> cd t
> ASAN_OPTIONS=detect_leaks=1 ./t0000-*.sh
>
> will confusingly fail to use UNLEAK().
>
> Your patch does the second one, but I think the first may be the
> least-bad option.
Thanks, I totally missed the fact that ASAN_OPTIONS was a runtime
thing. If we were to pursue this topic of enabling UNLEAK() outside
LSan, I agree the first would be necessary.
> But for leak-checking, we've already seen real cases where using LSan
> with higher optimization levels can lead to false positives (because the
> optimizer drops a value that is still in scope but not used in a code
> path that leads to exit()).
> ...
> So it may be that we really do want to keep leak-checking to "-O0
> -fsanitize=leak", and reserve "-O2 -fsanitize=address" for finding
> address bugs.
Yup, we have been burned a few times with this, IIRC.
^ permalink raw reply
* Re: [PATCH] Makefile: suppress annotated leaks with certain ASan options
From: Jeff King @ 2023-01-20 20:55 UTC (permalink / raw)
To: Taylor Blau
Cc: git, Derrick Stolee, Johannes Schindelin, Junio C Hamano,
Victoria Dye
In-Reply-To: <Y8r2Tn75g52YTIij@coredump.intra.peff.net>
On Fri, Jan 20, 2023 at 03:15:10PM -0500, Jeff King wrote:
> One problem UNLEAK() is that compile-time switch, but whether ASan does
> leak detection is a run-time choice. So you are stuck with either:
Er, this should be "one problem is that UNLEAK() is a compile-time
switch".
-Peff
^ permalink raw reply
* Re: [PATCH v6 04/12] test-http-server: add stub HTTP server test helper
From: Matthew John Cheetham @ 2023-01-20 22:05 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason,
Matthew John Cheetham via GitGitGadget
Cc: git, Derrick Stolee, Lessley Dennington, M Hickford,
Jeff Hostetler, Glen Choo, Victoria Dye
In-Reply-To: <230118.86k01kxfr7.gmgdl@evledraar.gmail.com>
On 2023-01-18 03:04, Ævar Arnfjörð Bjarmason wrote:
>
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
>
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>> [...]
>> +enum worker_result {
>> + /*
>> + * Operation successful.
>> + * Caller *might* keep the socket open and allow keep-alive.
>> + */
>> + WR_OK = 0,
>> [...]
>> + enum worker_result wr = WR_OK;
>> +
>> + if (client_addr)
>> + loginfo("Connection from %s:%s", client_addr, client_port);
>> +
>> + set_keep_alive(0, logerror);
>> +
>> + while (1) {
>> + if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
>> + logerror("unable to write response");
>> + wr = WR_IO_ERROR;
>> + }
>> +
>> + if (wr != WR_OK)
>> + break;
>> + }
>> +
>> + close(STDIN_FILENO);
>> + close(STDOUT_FILENO);
>> +
>> + return !!(wr & WR_IO_ERROR);
>> +}
>
> We have cases where we assign "0" to a bitfield-looking structure like
> this, but only in cases where we're planning to use it as a boolean too.
>
> Or, in other cases where we want some to be explicitly <-1.
>
> Here though we're adding a mixed "OK" and error use, which seems a bit
> odd. Shouldn't we pick one or the other?
You make a fair point about bitfields vs simple integer values. This was a
holdover from previous early hacking on this work where I had the bitfield
serve as a way to communicate the aspects of "does this count as an error?"
and "should we close the connection?".
Upon second thought, I think just simple integer values would be fine as
really only an "OK" and "HANGUP" are non-errors (the latter being the case
that the client gracefully ended the connection without an error and we
should exit).
Check for my next iteration for a rework on these `worker_result` values.
> So far (maybe in later commits?) nothing uses WR_HANGUP, and oddly we
> also use the bitfield-looking thing as a return value from main()....
We don't use the `enum worker_result` values as a return from `main`. We only
ever return 0 or 1 as we `return worker()` from `main`, and the only `return`
from `worker()` is `!!(wr & WR_IO_ERROR)` - 1 if we have `WR_IO_ERROR` set,
otherwise 0.
Thanks,
Matthew
^ permalink raw reply
* Re: [PATCH v6 05/12] test-http-server: add HTTP error response function
From: Matthew John Cheetham @ 2023-01-20 22:05 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason,
Matthew John Cheetham via GitGitGadget
Cc: git, Derrick Stolee, Lessley Dennington, M Hickford,
Jeff Hostetler, Glen Choo, Victoria Dye
In-Reply-To: <230118.86fsc8xffg.gmgdl@evledraar.gmail.com>
On 2023-01-18 03:07, Ævar Arnfjörð Bjarmason wrote:
>
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
>
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Introduce a function to the test-http-server test helper to write more
>> full and valid HTTP error responses, including all the standard response
>> headers like `Server` and `Date`.
>>
>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>> ---
>> t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++----
>> 1 file changed, 53 insertions(+), 5 deletions(-)
>>
>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
>> index 11071b1dd89..6cdac223a55 100644
>> --- a/t/helper/test-http-server.c
>> +++ b/t/helper/test-http-server.c
>> @@ -83,9 +83,59 @@ enum worker_result {
>> WR_HANGUP = 1<<1,
>> };
>
> ...okey, this is the commit that makes use of WR_HANGUP. Whatever else
> we do, let's then squash that addition into this change.
>
>> +static enum worker_result send_http_error(
>> + int fd,
>> + int http_code, const char *http_code_name,
>> + int retry_after_seconds, struct string_list *response_headers,
>> + enum worker_result wr_in)
>
> In general in this series you are mis-indenting argument lists. Our
> usual style is to wrap at 79 characters, then to align (with tabs and
> spaces) with the "(".
>
> So in this case:
>
> static enum worker_result send_http_error(int fd, int http_code,
> const char *http_code_name,
> int retry_after_seconds,
> struct string_list *response_headers,
> enum worker_result wr_in)
>
>> +{
>> + struct strbuf response_header = STRBUF_INIT;
>> + struct strbuf response_content = STRBUF_INIT;
>> + struct string_list_item *h;
>> + enum worker_result wr;
>> +
>> + strbuf_addf(&response_content, "Error: %d %s\r\n",
>> + http_code, http_code_name);
>
>
> Ditto here, where "http_code" should go on the preceding line...
>
>> + if (retry_after_seconds > 0)
>> + strbuf_addf(&response_content, "Retry-After: %d\r\n",
>> + retry_after_seconds);
>> +
>> + strbuf_addf (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
>
> ...and here there's a lack of such wrapping...
>
>> + strbuf_addstr(&response_header, "Cache-Control: private\r\n");
>> + strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
>> + strbuf_addf (&response_header, "Content-Length: %d\r\n", (int)response_content.len);
>> + if (retry_after_seconds > 0)
>> + strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
>> + strbuf_addf( &response_header, "Server: test-http-server/%s\r\n", git_version_string);
>> + strbuf_addf( &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
>
> ...here you're adding strange whitespace at the start of an argument list...
>
>> + if (response_headers)
>> + for_each_string_list_item(h, response_headers)
>> + strbuf_addf(&response_header, "%s\r\n", h->string);
>> + strbuf_addstr(&response_header, "\r\n");
Argh! Thanks again for catching these. I shall address them.
> To comment on the code a bit, this whole thing would be more readable
> IMO if your own headers were also a "struct string_list". Yes we'd waste
> a bit more memory, but in this case that's fine..
>
> I.e. don't add the "\r\n" every time, just:
>
> string_list_append(&headers, "Cache-Control: private");
>
> etc.
>
> Then at the end you'd do e.g.:
>
> add_headers(&buf, &headers);
> if (response_headers)
> add_headers(&buf, response_headers);
>
> Where the add_headers() is a trivial "static" helper which does that
> for_each_string_list_item() loop above.
In reality this only helps simplify the code in the case of a simple static
header like "Cache-Control: private". There's no `string_list_appendf` or
similar where I need to append a header that contains dynamic information
(date, content length, etc).
Building the `strbuf` directly, and specifying the CRLF seems a lot easier IMO.
>> while (1) {
>> - if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
>> - logerror("unable to write response");
>> - wr = WR_IO_ERROR;
>> - }
>> + wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
>> + NULL, WR_OK | WR_HANGUP);
>
> This *does* use correct wrapping & indenation for a continuing argument
> list.
Thanks,
Matthew
^ permalink raw reply
* Re: [PATCH v6 06/12] test-http-server: add HTTP request parsing
From: Matthew John Cheetham @ 2023-01-20 22:05 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason,
Matthew John Cheetham via GitGitGadget
Cc: git, Derrick Stolee, Lessley Dennington, M Hickford,
Jeff Hostetler, Glen Choo, Victoria Dye
In-Reply-To: <230118.86bkmwxf6e.gmgdl@evledraar.gmail.com>
On 2023-01-18 03:14, Ævar Arnfjörð Bjarmason wrote:
>
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
>
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>>
>> Add ability to parse HTTP requests to the test-http-server test helper.
>>
>> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
>> ---
>> t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++-
>> 1 file changed, 173 insertions(+), 2 deletions(-)
>>
>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
>> index 6cdac223a55..36f4a54fe6d 100644
>> --- a/t/helper/test-http-server.c
>> +++ b/t/helper/test-http-server.c
>> @@ -83,6 +83,42 @@ enum worker_result {
>> WR_HANGUP = 1<<1,
>> };
>>
>> +/*
>> + * Fields from a parsed HTTP request.
>> + */
>> +struct req {
>> + struct strbuf start_line;
>> +
>> + const char *method;
>> + const char *http_version;
>> +
>> + struct strbuf uri_path;
>> + struct strbuf query_args;
>> +
>> + struct string_list header_list;
>> + const char *content_type;
>> + ssize_t content_length;
>> +};
>> +
>> +#define REQ__INIT { \
>> + .start_line = STRBUF_INIT, \
>> + .uri_path = STRBUF_INIT, \
>> + .query_args = STRBUF_INIT, \
>> + .header_list = STRING_LIST_INIT_NODUP, \
>> + .content_type = NULL, \
>> + .content_length = -1 \
>> + }
>
> Style nit: Don't indent the trailing "}", and add a "," after the last
> "content_length" item.
>
> We omit the comma by convention when there really should not be another
> item, such as when we have a "NULL" terminator, here though we might add
> a struct element at the end, so...
Sure.
>> +static enum worker_result req__read(struct req *req, int fd)
>> +{
>> + struct strbuf h = STRBUF_INIT;
>> + struct string_list start_line_fields = STRING_LIST_INIT_DUP;
>> + int nr_start_line_fields;
>> + const char *uri_target;
>> + const char *query;
>> + char *hp;
>> + const char *hv;
>> +
>> + enum worker_result result = WR_OK;
>> +
>> + /*
>> + * Read line 0 of the request and split it into component parts:
>> + *
>> + * <method> SP <uri-target> SP <HTTP-version> CRLF
>> + *
>> + */
>> + if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
>> + result = WR_OK | WR_HANGUP;
>> + goto done;
>> + }
>> +
>> + strbuf_trim_trailing_newline(&req->start_line);
>> +
>> + nr_start_line_fields = string_list_split(&start_line_fields,
>> + req->start_line.buf,
>> + ' ', -1);
>> + if (nr_start_line_fields != 3) {
>> + logerror("could not parse request start-line '%s'",
>> + req->start_line.buf);
>> + result = WR_IO_ERROR;
>> + goto done;
>> + }
>> +
>> + req->method = xstrdup(start_line_fields.items[0].string);
>> + req->http_version = xstrdup(start_line_fields.items[2].string);
>> +
>> + uri_target = start_line_fields.items[1].string;
>> +
>> + if (strcmp(req->http_version, "HTTP/1.1")) {
>> + logerror("unsupported version '%s' (expecting HTTP/1.1)",
>> + req->http_version);
>> + result = WR_IO_ERROR;
>> + goto done;
>> + }
>> +
>> + query = strchr(uri_target, '?');
>> +
>> + if (query) {
>> + strbuf_add(&req->uri_path, uri_target, (query - uri_target));
>> + strbuf_trim_trailing_dir_sep(&req->uri_path);
>> + strbuf_addstr(&req->query_args, query + 1);
>> + } else {
>> + strbuf_addstr(&req->uri_path, uri_target);
>> + strbuf_trim_trailing_dir_sep(&req->uri_path);
>> + }
>> +
>> + /*
>> + * Read the set of HTTP headers into a string-list.
>> + */
>> + while (1) {
>> + if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
>> + goto done;
>> + strbuf_trim_trailing_newline(&h);
>> +
>> + if (!h.len)
>> + goto done; /* a blank line ends the header */
>> +
>> + hp = strbuf_detach(&h, NULL);
>> + string_list_append(&req->header_list, hp);
>> +
>> + /* also store common request headers as struct req members */
>> + if (skip_prefix(hp, "Content-Type: ", &hv)) {
>> + req->content_type = hv;
>> + } else if (skip_prefix(hp, "Content-Length: ", &hv)) {
>> + req->content_length = strtol(hv, &hp, 10);
>
> In POSIX the "ssize_t" is not a "this is the unsigned size_t", but can
> be a much smaller integer type (although in practice it tends to be the
> signed version of "size_t".
>
> But this seems like a potential overflow trap as a result, but sometimes
> we need to live with "ssize_t".
>
> However, in this case it seems like we don't, as it seems the only
> reason you init'd this to -1 and then...
>
>> + if (trace2_is_enabled()) {
>> + struct string_list_item *item;
>> + trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
>> + trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
>> + trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
>> + trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
>> + trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
>> + if (req->content_length >= 0)
>> + trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
>
> ...use that ">= 0" is to keep the state of "did I assign to this above?
>
> So firstly, shouldn't we error or something on a "Content-Length: 0",
> and aside from that wouldn't we just have a "int have_content_length =
> 0" in this function that we'd then flip to 1?
It seems like the perfect type for such a non-zero-or-error size value; from
POSIX specifications[1]:
> ...
> size_t
> Used for sizes of objects.
> ssize_t
> Used for a count of bytes or an error indication.
> ...
But you're probably right here that `ssize_t` isn't that suitable in practice
due to the comically low minimum size of the `SSIZE_MAX` (2^15 I believe).
RFC 9110 §8.6 [2] addresses the `Content-Length` HTTP header and says that its
value should be non-negative, but also have no upper bound; we're gonna have to
set at least some practical limit.
Libcurl handles this by writing it's own parsing function that's good up to
a max 64-bit integer value [3][4].
Given this is for a test helper and only going to be receiving data from tests,
I propose just using something like `uintmax_t` and storing a bit with
`unsigned has_content_length:1;` to show if we actually got a header in the
request or not.
Thanks,
Matthew
[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_types.h.html
[2] https://www.rfc-editor.org/rfc/rfc9110.html#section-8.6
[3] https://github.com/curl/curl/blob/6113dec2a829d4ab766428ccca9535b7a5efd012/lib/http.c#L3348-L3349
[4] https://github.com/curl/curl/blob/6113dec2a829d4ab766428ccca9535b7a5efd012/lib/strtoofft.c#L214-L218
^ permalink raw reply
* Re: [PATCH v6 08/12] test-http-server: add simple authentication
From: Matthew John Cheetham @ 2023-01-20 22:05 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason,
Matthew John Cheetham via GitGitGadget
Cc: git, Derrick Stolee, Lessley Dennington, M Hickford,
Jeff Hostetler, Glen Choo, Victoria Dye
In-Reply-To: <230118.867cxkxece.gmgdl@evledraar.gmail.com>
On 2023-01-18 03:21, Ævar Arnfjörð Bjarmason wrote:
>
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
>
>
>> +static struct auth_module *get_auth_module(const char *scheme, int create)
>> +{
>> + int i;
>> + struct auth_module *mod;
>> + for (i = 0; i < auth_modules_nr; i++) {
>
> We can use "for (size_t i = 0" syntax now, let's do that here to not mix
> "size_t" and "int" types needlessly.
Yep!
>> + if (create) {
>> + struct auth_module *mod = xmalloc(sizeof(struct auth_module));
>> + mod->scheme = xstrdup(scheme);
>> + mod->challenge_params = NULL;
>> + CALLOC_ARRAY(mod->tokens, 1);
>> + string_list_init_dup(mod->tokens);
>
> Don't use CALLOC_ARRAY() if you're then going to use
> string_list_init_dup() (which is good!), use ALLOC_ARRAY() instead. We
> don't need to set the memory to 0, only to overwrite it entirely again.
Sure.
>> + ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
>> + auth_modules[auth_modules_nr++] = mod;
>
> I have not looked at the whole context here, but instead of:
>
> struct auth_module {
> char *scheme;
> char *challenge_params;
> struct string_list *tokens;
> };
>
> Why not:
>
> struct auth_module {
> char *challenge_params;
> struct string_list *tokens;
> };
>
> Then you could use a "struct string_list" for this, make the "scheme" be
> the "string" member, and stick the remaining two fields in the "util",
> and thus save yourself the manual memory management etc.
I looked at this, but this then means being more careful when looping over
different `struct auth_module`s to keep the current 'scheme' and `*mod` in
sync/together. Just feels like overkill right now.
>> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)
>> +{
>> + enum auth_result result = AUTH_UNKNOWN;
>> + struct string_list hdrs = STRING_LIST_INIT_NODUP;
>> + struct auth_module *mod;
>> +
>> + struct string_list_item *hdr;
>> + struct string_list_item *token;
>> + const char *v;
>> + struct strbuf **split = NULL;
>> + int i;
>> + char *challenge;
>> +
>> + /*
>> + * Check all auth modules and try to validate the request.
>> + * The first Authorization header that matches a known auth module
>> + * scheme will be consulted to either approve or deny the request.
>> + * If no module is found, or if there is no valid token, then 401 error.
>> + * Otherwise, only permit the request if anonymous auth is enabled.
>> + * It's atypical for user agents/clients to send multiple Authorization
>> + * headers, but not explicitly forbidden or defined.
>> + */
>> + for_each_string_list_item(hdr, &req->header_list) {
>> + if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
>> + split = strbuf_split_str(v, ' ', 2);
>> + if (!split[0] || !split[1]) continue;
>> +
>> + /* trim trailing space ' ' */
>> + strbuf_setlen(split[0], split[0]->len - 1);
>> +
>> + mod = get_auth_module(split[0]->buf, 0);
>> + if (mod) {
>> + result = AUTH_DENY;
>> +
>> + for_each_string_list_item(token, mod->tokens) {
>> + if (!strcmp(split[1]->buf, token->string)) {
>> + result = AUTH_ALLOW;
>> + break;
>> + }
>> + }
>> +
>> + goto done;
>
> Sometimes we need a strbuf_split_str, but in this case couldn't you use
> the in-place "struct string_list" variant of that instead, and just
> carry a "size_t len" here for it, which you'd then pass to
> get_auth_module() (which this commit adds)?
`get_auth_module` taking a scheme name as parameter is a more sensible, IMO,
than a `string_list` or `string_list_item` and an offset. Given this is a test
helper, performance also isn't a priority. Readability wins here I think.
> Also, you "split" in the loop, but...
>
>> + strbuf_list_free(split);
> ...only free() the last one here, isn't this leaking?
Yes, it is. Will fix in next iteration.
>> +static int split_auth_param(const char *str, char **scheme, char **val)
>> +{
>> + struct strbuf **p = strbuf_split_str(str, ':', 2);
>> +
>> + if (!p[0])
>> + return -1;
>> +
>> + /* trim trailing ':' */
>> + if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')
>
> Don't compare unsigned length fields to "> 0", just do "if (len &&
> ....)".
Sure!
> Also, maybe I'm just groggy today, but how do we have a trailing ":" if
> we just split on ":", and with a limit such that...
>
>> + if (p[1])
>> + *val = strbuf_detach(p[1], NULL);
>
> ...we have an item after that?
Because that's how the `strbuf_split_str` function works. The comments
in the header file even call that out. "The substrings include the
terminator". From strbuf.h:
/**
* Split str (of length slen) at the specified terminator character.
* Return a null-terminated array of pointers to strbuf objects
* holding the substrings. The substrings include the terminator,
* except for the last substring, which might be unterminated if the
* original string did not end with a terminator. [cut] ...
...
*/
struct strbuf **strbuf_split_buf(const char *str, size_t len,
int terminator, int max);
>> +static int read_auth_config(const char *name, const char *val, void *data)
>> +{
>> + int ret = 0;
>> + char *scheme = NULL;
>
> Don't init this to NULL, instead the split_auth_param() return value
> should be trusted, the compiler will then help us catch errors, no?
>
>> + char *token = NULL;
>> + char *challenge = NULL;
>
> In this case it *is* needed though, as the function will return
> non-errors, but *maybe* give us the second out parameter.
>
> For such a function though, isn't just assigning "*second_param = NULL"
> at the start of it less of a "running with scissors" pattern?
>
>> + struct auth_module *mod = NULL;
>
> This NULL assignment can be dropped, we assign to it below
> unconditionally before using it.
All of these variables need to be initialised to NULL because not all
arms of the `if-elseif` chain assign to all of these variables, but
we always `free` all of them at the function exit.
For example,
char *scheme = NULL;
char *token = NULL;
char *challenge = NULL;
...
} else if (!strcmp(name, "auth.allowanonymous")) {
allow_anonymous = git_config_bool(name, val);
} else {
...
free(scheme);
free(token);
free(challenge);
>> +
>> + if (!strcmp(name, "auth.challenge")) {
>> + if (split_auth_param(val, &scheme, &challenge)) {
>> + ret = error("invalid auth challenge '%s'", val);
>> + goto cleanup;
>> + }
>> +
>> + mod = get_auth_module(scheme, 1);
>> +
>> + /* Replace any existing challenge parameters */
>> + free(mod->challenge_params);
>> + mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
>> + } else if (!strcmp(name, "auth.token")) {
>> + if (split_auth_param(val, &scheme, &token)) {
>> + ret = error("invalid auth token '%s'", val);
>> + goto cleanup;
>> + }
>> +
>> + mod = get_auth_module(scheme, 1);
>> +
>> + /*
>> + * Append to set of valid tokens unless an empty token value
>> + * is provided, then clear the existing list.
>> + */
>> + if (token)
>> + string_list_append(mod->tokens, token);
>> + else
>> + string_list_clear(mod->tokens, 1);
>> + } else if (!strcmp(name, "auth.allowanonymous")) {
>> + allow_anonymous = git_config_bool(name, val);
>> + } else {
>> + warning("unknown auth config '%s'", name);
>> + }
>> +
>> +cleanup:
>> + free(scheme);
>> + free(token);
>> + free(challenge);
>> +
>> + return ret;
>> +}
>> +
Thanks,
Matthew
^ permalink raw reply
* Re: [PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers
From: Matthew John Cheetham @ 2023-01-20 22:05 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason,
Matthew John Cheetham via GitGitGadget
Cc: git, Derrick Stolee, Lessley Dennington, M Hickford,
Jeff Hostetler, Glen Choo, Victoria Dye
In-Reply-To: <230118.86y1q0vzhh.gmgdl@evledraar.gmail.com>
On 2023-01-18 03:42, Ævar Arnfjörð Bjarmason wrote:
>
> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:
>
>> From: Matthew John Cheetham <mjcheetham@outlook.com>
>
>> + strbuf_add(&buf, ptr, size);
>> +
>> + /* Strip the CRLF that should be present at the end of each field */
>> + strbuf_trim_trailing_newline(&buf);
>> +
>> + /* Start of a new WWW-Authenticate header */
>> + if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
>> + while (isspace(*val))
>> + val++;
>> +
>> + strvec_push(values, val);
>> + http_auth.header_is_last_match = 1;
>> + goto exit;
>> [...]
>> + if (http_auth.header_is_last_match && isspace(*buf.buf)) {
>> + /* Trim leading whitespace from this continuation hdr line. */
>> + strbuf_ltrim(&buf);
>
>
> The mixture of this isspace() loop and then strbuf_ltrim() seems odd,
> why not stick with the strbuf API?
>
> I.e. after skip_iprefix() strbuf_splice() the start of the string away,
> then use strbuf_ltrim() in the first "if" branch here?
You mean like this?
size_t size = st_mult(eltsize, nmemb);
struct strvec *values = &http_auth.wwwauth_headers;
struct strbuf buf = STRBUF_INIT;
- const char *val;
/*
* Header lines may not come NULL-terminated from libcurl so we must
@@ -216,11 +215,11 @@ static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
strbuf_trim_trailing_newline(&buf);
/* Start of a new WWW-Authenticate header */
- if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
- while (isspace(*val))
- val++;
+ if (istarts_with(buf.buf, "www-authenticate:")) {
+ strbuf_splice(&buf, 0, 17, NULL, 0);
+ strbuf_ltrim(&buf);
- strvec_push(values, val);
+ strvec_push(values, buf.buf);
http_auth.header_is_last_match = 1;
goto exit;
}
I don't particularly like this given we're now introducing the 'magic' number
17 that's the length of `www-authenticate:`, plus `strbuf_splice` is doing
a lot more work moving pieces of memory around rather than just producing
a new starting pointer to the start of the value (skipping leading whitespace).
> Likewise this is open-coding the "isspace" in strbuf_ltrim() for the
> second "if". Maybe run the strbuf_ltrim() unconditionally, save away the
> length before, and then:
>
> if (http_auth.header_is_last_match && prev_len != buf.len) { ...
>
> ?
The suggestion of trimming and comparing lengths just makes a piece of code
handling a little-known edge case less immediately obvious in its intent in
my opinion. The current implementation of "if starts with a single space"
matches the definition of continuation header lines, rather than re-piecing
together this from "why are we trimming and comparing lengths?"
Perf-wise the current implementation is only adding one extra `isspace`
call which we're potentially about to do in a loop inside of `strbuf_ltrim`
anyway. Plus, the common case will be a single space anyway.
Thanks,
Matthew
^ permalink raw reply
* [PATCH v7 00/12] Enhance credential helper protocol to include auth headers
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham
In-Reply-To: <pull.1352.v6.git.1674012618.gitgitgadget@gmail.com>
Following from my original RFC submission [0], this submission is considered
ready for full review. This patch series is now based on top of current
master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now
separately submitted patches [1] to fix up the other credential helpers'
behaviour.
In this patch series I update the existing credential helper design in order
to allow for some new scenarios, and future evolution of auth methods that
Git hosts may wish to provide. I outline the background, summary of changes
and some challenges below.
Testing these new additions, I introduce a new test helper test-http-server
that acts as a frontend to git-http-backend; a mini HTTP server sharing code
with git-daemon, with simple authentication configurable by a config file.
Background
==========
Git uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and
Git. Here I focus on the Smart HTTP protocol, and attempt to enhance the
authentication capabilities of this protocol to address limitations (see
below).
The Smart HTTP protocol in Git supports a few different types of HTTP
authentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)
[4]. Git uses a extensible model where credential helpers can provide
credentials for protocols [5]. Several helpers support alternatives such as
OAuth authentication (RFC 6749) [6], but this is typically done as an
extension. For example, a helper might use basic auth and set the password
to an OAuth Bearer access token. Git uses standard input and output to
communicate with credential helpers.
After a HTTP 401 response, Git would call a credential helper with the
following over standard input:
protocol=https
host=example.com
And then a credential helper would return over standard output:
protocol=https
host=example.com
username=bob@id.example.com
password=<BEARER-TOKEN>
Git then the following request to the remote, including the standard HTTP
Authorization header (RFC 7235 Section 4.2) [7]:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: Basic base64(bob@id.example.com:<BEARER-TOKEN>)
Credential helpers are encouraged (see gitcredentials.txt) to return the
minimum information necessary.
Limitations
===========
Because this credential model was built mostly for password based
authentication systems, it's somewhat limited. In particular:
1. To generate valid credentials, additional information about the request
(or indeed the requestee and their device) may be required. For example,
OAuth is based around scopes. A scope, like "git.read", might be
required to read data from the remote. However, the remote cannot tell
the credential helper what scope is required for this request.
2. This system is not fully extensible. Each time a new type of
authentication (like OAuth Bearer) is invented, Git needs updates before
credential helpers can take advantage of it (or leverage a new
capability in libcurl).
Goals
=====
* As a user with multiple federated cloud identities:
* Reach out to a remote and have my credential helper automatically
prompt me for the correct identity.
* Allow credential helpers to differentiate between different authorities
or authentication/authorization challenge types, even from the same DNS
hostname (and without needing to use credential.useHttpPath).
* Leverage existing authentication systems built-in to many operating
systems and devices to boost security and reduce reliance on passwords.
* As a Git host and/or cloud identity provider:
* Enforce security policies (like requiring two-factor authentication)
dynamically.
* Allow integration with third party standard based identity providers in
enterprises allowing customers to have a single plane of control for
critical identities with access to source code.
Design Principles
=================
* Use the existing infrastructure. Git credential helpers are an
already-working model.
* Follow widely-adopted time-proven open standards, avoid net new ideas in
the authentication space.
* Minimize knowledge of authentication in Git; maintain modularity and
extensibility.
Proposed Changes
================
1. Teach Git to read HTTP response headers, specifically the standard
WWW-Authenticate (RFC 7235 Section 4.1) headers.
2. Teach Git to include extra information about HTTP responses that require
authentication when calling credential helpers. Specifically the
WWW-Authenticate header information.
Because the extra information forms an ordered list, and the existing
credential helper I/O format only provides for simple key=value pairs,
we introduce a new convention for transmitting an ordered list of
values. Key names that are suffixed with a C-style array syntax should
have values considered to form an order list, i.e. key[]=value, where
the order of the key=value pairs in the stream specifies the order.
For the WWW-Authenticate header values we opt to use the key wwwauth[].
Handling the WWW-Authenticate header in detail
==============================================
RFC 6750 [8] envisions that OAuth Bearer resource servers would give
responses that include WWW-Authenticate headers, for example:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
Specifically, a WWW-Authenticate header consists of a scheme and arbitrary
attributes, depending on the scheme. This pattern enables generic OAuth or
OpenID Connect [9] authorities. Note that it is possible to have several
WWW-Authenticate challenges in a response.
First Git attempts to make a request, unauthenticated, which fails with a
401 response and includes WWW-Authenticate header(s).
Next, Git invokes a credential helper which may prompt the user. If the user
approves, a credential helper can generate a token (or any auth challenge
response) to be used for that request.
For example: with a remote that supports bearer tokens from an OpenID
Connect [9] authority, a credential helper can use OpenID Connect's
Discovery [10] and Dynamic Client Registration [11] to register a client and
make a request with the correct permissions to access the remote. In this
manner, a user can be dynamically sent to the right federated identity
provider for a remote without any up-front configuration or manual
processes.
Following from the principle of keeping authentication knowledge in Git to a
minimum, we modify Git to add all WWW-Authenticate values to the credential
helper call.
Git sends over standard input:
protocol=https
host=example.com
wwwauth[]=Bearer realm="login.example", scope="git.readwrite"
wwwauth[]=Basic realm="login.example"
A credential helper that understands the extra wwwauth[n] property can
decide on the "best" or correct authentication scheme, generate credentials
for the request, and interact with the user.
The credential helper would then return over standard output:
protocol=https
host=example.com
path=foo.git
username=bob@identity.example
password=<BEARER-TOKEN>
Note that WWW-Authenticate supports multiple challenges, either in one
header:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
or in multiple headers:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="login.example", scope="git.readwrite"
WWW-Authenticate: Basic realm="login.example"
These have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the
implementation, Git will not merge or split up any of these WWW-Authenticate
headers, and instead pass each header line as one credential helper
property. The credential helper is responsible for splitting, merging, and
otherwise parsing these header values.
An alternative option to sending the header fields individually would be to
merge the header values in to one key=value property, for example:
...
wwwauth=Bearer realm="login.example", scope="git.readwrite", Basic realm="login.example"
Future work
===========
In the future we can further expand the protocol to allow credential helpers
decide the best authentication scheme. Today credential helpers are still
only expected to return a username/password pair to Git, meaning the other
authentication schemes that may be offered still need challenge responses
sent via a Basic Authorization header. The changes outlined above still
permit helpers to select and configure an available authentication mode, but
require the remote for example to unpack a bearer token from a basic
challenge.
More careful consideration is required in the handling of custom
authentication schemes which may not have a username, or may require
arbitrary additional request header values be set.
For example imagine a new "FooBar" authentication scheme that is surfaced in
the following response:
HTTP/1.1 401 Unauthorized
WWW-Authenticate: FooBar realm="login.example", algs="ES256 PS256"
With support for arbitrary authentication schemes, Git would call credential
helpers with the following over standard input:
protocol=https
host=example.com
wwwauth[]=FooBar realm="login.example", algs="ES256 PS256", nonce="abc123"
And then an enlightened credential helper could return over standard output:
protocol=https
host=example.com
authtype=FooBar
username=bob@id.example.com
password=<FooBar credential>
header[]=X-FooBar: 12345
header[]=X-FooBar-Alt: ABCDEF
Git would be expected to attach this authorization header to the next
request:
GET /info/refs?service=git-upload-pack HTTP/1.1
Host: git.example
Git-Protocol: version=2
Authorization: FooBar <FooBar credential>
X-FooBar: 12345
X-FooBar-Alt: ABCDEF
Why not SSH?
============
There's nothing wrong with SSH. However, Git's Smart HTTP transport is
widely used, often with OAuth Bearer tokens. Git's Smart HTTP transport
sometimes requires less client setup than SSH transport, and works in
environments when SSH ports may be blocked. As long as Git supports HTTP
transport, it should support common and popular HTTP authentication methods.
References
==========
* [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth
headers
https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/
* [1] [PATCH 0/3] Correct credential helper discrepancies handling input
https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/
* [2] Git on the Server - The Protocols
https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols
* [3] HTTP Authentication: Basic and Digest Access Authentication
https://datatracker.ietf.org/doc/html/rfc2617
* [4] The Simple and Protected GSS-API Negotiation Mechanism
https://datatracker.ietf.org/doc/html/rfc2478
* [5] Git Credentials - Custom Helpers
https://git-scm.com/docs/gitcredentials#_custom_helpers
* [6] The OAuth 2.0 Authorization Framework
https://datatracker.ietf.org/doc/html/rfc6749
* [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication
https://datatracker.ietf.org/doc/html/rfc7235
* [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage
https://datatracker.ietf.org/doc/html/rfc6750
* [9] OpenID Connect Core 1.0
https://openid.net/specs/openid-connect-core-1_0.html
* [10] OpenID Connect Discovery 1.0
https://openid.net/specs/openid-connect-discovery-1_0.html
* [11] OpenID Connect Dynamic Client Registration 1.0
https://openid.net/specs/openid-connect-registration-1_0.html
* [12] Hypertext Transfer Protocol (HTTP/1.1)
https://datatracker.ietf.org/doc/html/rfc2616
Updates from RFC
================
* Submitted first three patches as separate submission:
https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/
* Various style fixes and updates to- and addition of comments.
* Drop the explicit integer index in new 'array' style credential helper
attrbiutes ("key[n]=value" becomes just "key[]=value").
* Added test helper; a mini HTTP server, and several tests.
Updates in v3
=============
* Split final patch that added the test-http-server in to several, easier
to review patches.
* Updated wording in git-credential.txt to clarify which side of the
credential helper protocol is sending/receiving the new wwwauth and
authtype attributes.
Updates in v4
=============
* Drop authentication scheme selection authtype attribute patches to
greatly simplify the series; auth scheme selection is punted to a future
series. This series still allows credential helpers to generate
credentials and intelligently select correct identities for a given auth
challenge.
Updates in v5
=============
* Libify parts of daemon.c and share implementation with test-http-server.
* Clarify test-http-server Git request regex pattern and auth logic
comments.
* Use STD*_FILENO in place of 'magic' file descriptor numbers.
* Use strbuf_* functions in continuation header parsing.
* Use configuration file to configure auth for test-http-server rather than
command-line arguments. Add ability to specify arbitrary extra headers
that is useful for testing 'malformed' server responses.
* Use st_mult over unchecked multiplication in http.c curl callback
functions.
* Fix some documentation line break issues.
* Reorder some commits to bring in the tests and test-http-server helper
first and, then the WWW-Authentication changes, alongside tests to cover.
* Expose previously static strvec_push_nodup function.
* Merge the two timeout args for test-http-server (--timeout and
--init-timeout) that were a hang-over from the original daemon.c but are
no longer required here.
* Be more careful around continuation headers where they may be empty
strings. Add more tests to cover these header types.
* Include standard trace2 tracing calls at start of test-http-server
helper.
Updates in v6
=============
* Clarify the change to make logging optional in the check_dead_children()
function during libification of daemon.c.
* Fix missing pointer dereference bugs identified in libification of child
process handling functions for daemon.c.
* Add doc comments to child process handling function declarations in the
daemon-utils.h header.
* Align function parameter names with variable names at callsites for
libified daemon functions.
* Re-split out the test-http-server test helper commits in to smaller
patches: error response handling, request parsing, http-backend
pass-through, simple authentication, arbitrary header support.
* Call out auth configuration file format for test-http-server test helper
and supported options in commit messages, as well as a test to exercise
and demonstrate these options.
* Permit auth.token and auth.challenge to appear in any order; create the
struct auth_module just-in-time as options for that scheme are read. This
simplifies the configuration authoring of the test-http-server test
helper.
* Update tests to use auth.allowAnoymous in the patch that introduces the
new test helper option.
* Drop the strvec_push_nodup() commit and update the implementation of HTTP
request header line folding to use xstrdup and strvec_pop and _pushf.
* Use size_t instead of int in credential.c when iterating over the struct
strvec credential members. Also drop the not required const and cast from
the full_key definition and free.
* Replace in-tree test-credential-helper-reply.sh test cred helper script
with the lib-credential-helper.sh reusable 'lib' test script and shell
functions to configure the helper behaviour.
* Leverage sed over the while read $line loop in the test credential helper
script.
Updates in v7
=============
* Address several whitespace and arg/param list alignment issues.
* Rethink the test-http-helper worker-mode error and result enum to be more
simple and more informative to the nature of the error.
* Use uintmax_t to store the Content-Length of a request in the helper
test-http-server. Maintain a bit flag to store if we received such a
header.
* Return a "400 Bad Request" HTTP response if we fail to parse the request
in the test-http-server.
* Add test case to cover request message parsing in test-http-server.
* Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in
get_auth_module.
* Correctly free the split strbufs created in the header parsing loop in
test-http-server.
* Avoid needless comparison > 0 for unsigned types.
* Always set optional outputs to NULL if not present in test helper config
value handling.
* Remove an accidentally commented-out test cleanup line for one test case
in t5556.
Matthew John Cheetham (12):
daemon: libify socket setup and option functions
daemon: libify child process handling functions
daemon: rename some esoteric/laboured terminology
test-http-server: add stub HTTP server test helper
test-http-server: add HTTP error response function
test-http-server: add HTTP request parsing
test-http-server: pass Git requests to http-backend
test-http-server: add simple authentication
test-http-server: add sending of arbitrary headers
http: replace unsafe size_t multiplication with st_mult
http: read HTTP WWW-Authenticate response headers
credential: add WWW-Authenticate header to cred requests
Documentation/git-credential.txt | 19 +-
Makefile | 2 +
contrib/buildsystems/CMakeLists.txt | 11 +-
credential.c | 12 +
credential.h | 15 +
daemon-utils.c | 286 +++++++++
daemon-utils.h | 55 ++
daemon.c | 306 +--------
http.c | 98 ++-
t/helper/.gitignore | 1 +
t/helper/test-http-server.c | 943 ++++++++++++++++++++++++++++
t/lib-credential-helper.sh | 27 +
t/t5556-http-auth.sh | 463 ++++++++++++++
13 files changed, 1936 insertions(+), 302 deletions(-)
create mode 100644 daemon-utils.c
create mode 100644 daemon-utils.h
create mode 100644 t/helper/test-http-server.c
create mode 100644 t/lib-credential-helper.sh
create mode 100755 t/t5556-http-auth.sh
base-commit: c48035d29b4e524aed3a32f0403676f0d9128863
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v7
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v7
Pull-Request: https://github.com/gitgitgadget/git/pull/1352
Range-diff vs v6:
1: 74b0de14185 = 1: 74b0de14185 daemon: libify socket setup and option functions
2: b6ba344a671 = 2: b6ba344a671 daemon: libify child process handling functions
3: 9967401c972 = 3: 9967401c972 daemon: rename some esoteric/laboured terminology
4: d6e5e8825e8 ! 4: 17c890ee108 test-http-server: add stub HTTP server test helper
@@ t/helper/test-http-server.c (new)
+
+/*
+ * The code in this section is used by "worker" instances to service
-+ * a single connection from a client. The worker talks to the client
-+ * on 0 and 1.
++ * a single connection from a client. The worker talks to the client
++ * on stdin and stdout.
+ */
+
+enum worker_result {
@@ t/helper/test-http-server.c (new)
+ * Operation successful.
+ * Caller *might* keep the socket open and allow keep-alive.
+ */
-+ WR_OK = 0,
++ WR_OK = 0,
+
+ /*
-+ * Various errors while processing the request and/or the response.
++ * Fatal error that is not recoverable.
+ * Close the socket and clean up.
+ * Exit child-process with non-zero status.
+ */
-+ WR_IO_ERROR = 1<<0,
-+
-+ /*
-+ * Close the socket and clean up. Does not imply an error.
-+ */
-+ WR_HANGUP = 1<<1,
++ WR_FATAL_ERROR = 1,
+};
+
+static enum worker_result worker(void)
@@ t/helper/test-http-server.c (new)
+ while (1) {
+ if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
+ logerror("unable to write response");
-+ wr = WR_IO_ERROR;
++ wr = WR_FATAL_ERROR;
+ }
+
+ if (wr != WR_OK)
@@ t/helper/test-http-server.c (new)
+ close(STDIN_FILENO);
+ close(STDOUT_FILENO);
+
-+ return !!(wr & WR_IO_ERROR);
++ /* Only WR_OK should result in a non-zero exit code */
++ return wr != WR_OK;
+}
+
+static int max_connections = 32;
5: 79805f042b9 ! 5: 6e70e304cfe test-http-server: add HTTP error response function
@@ Commit message
## t/helper/test-http-server.c ##
@@ t/helper/test-http-server.c: enum worker_result {
- WR_HANGUP = 1<<1,
+ * Exit child-process with non-zero status.
+ */
+ WR_FATAL_ERROR = 1,
++
++ /*
++ * Close the socket and clean up. Does not imply an error.
++ */
++ WR_HANGUP = 2,
};
-+static enum worker_result send_http_error(
-+ int fd,
-+ int http_code, const char *http_code_name,
-+ int retry_after_seconds, struct string_list *response_headers,
-+ enum worker_result wr_in)
++static enum worker_result send_http_error(int fd, int http_code,
++ const char *http_code_name,
++ int retry_after_seconds,
++ struct string_list *response_headers,
++ enum worker_result wr_in)
+{
+ struct strbuf response_header = STRBUF_INIT;
+ struct strbuf response_content = STRBUF_INIT;
+ struct string_list_item *h;
+ enum worker_result wr;
+
-+ strbuf_addf(&response_content, "Error: %d %s\r\n",
-+ http_code, http_code_name);
++ strbuf_addf(&response_content, "Error: %d %s\r\n", http_code,
++ http_code_name);
++
+ if (retry_after_seconds > 0)
+ strbuf_addf(&response_content, "Retry-After: %d\r\n",
+ retry_after_seconds);
+
-+ strbuf_addf (&response_header, "HTTP/1.1 %d %s\r\n", http_code, http_code_name);
++ strbuf_addf(&response_header, "HTTP/1.1 %d %s\r\n", http_code,
++ http_code_name);
+ strbuf_addstr(&response_header, "Cache-Control: private\r\n");
-+ strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
-+ strbuf_addf (&response_header, "Content-Length: %d\r\n", (int)response_content.len);
++ strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
++ strbuf_addf(&response_header, "Content-Length: %"PRIuMAX"\r\n",
++ (uintmax_t)response_content.len);
++
+ if (retry_after_seconds > 0)
-+ strbuf_addf(&response_header, "Retry-After: %d\r\n", retry_after_seconds);
-+ strbuf_addf( &response_header, "Server: test-http-server/%s\r\n", git_version_string);
-+ strbuf_addf( &response_header, "Date: %s\r\n", show_date(time(NULL), 0, DATE_MODE(RFC2822)));
++ strbuf_addf(&response_header, "Retry-After: %d\r\n",
++ retry_after_seconds);
++
++ strbuf_addf(&response_header, "Server: test-http-server/%s\r\n",
++ git_version_string);
++ strbuf_addf(&response_header, "Date: %s\r\n", show_date(time(NULL), 0,
++ DATE_MODE(RFC2822)));
++
+ if (response_headers)
+ for_each_string_list_item(h, response_headers)
+ strbuf_addf(&response_header, "%s\r\n", h->string);
@@ t/helper/test-http-server.c: enum worker_result {
+
+ if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
+ logerror("unable to write response header");
-+ wr = WR_IO_ERROR;
++ wr = WR_FATAL_ERROR;
+ goto done;
+ }
+
+ if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
+ logerror("unable to write response content body");
-+ wr = WR_IO_ERROR;
++ wr = WR_FATAL_ERROR;
+ goto done;
+ }
+
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
while (1) {
- if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
- logerror("unable to write response");
-- wr = WR_IO_ERROR;
+- wr = WR_FATAL_ERROR;
- }
+ wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
-+ NULL, WR_OK | WR_HANGUP);
++ NULL, WR_HANGUP);
if (wr != WR_OK)
break;
+@@ t/helper/test-http-server.c: static enum worker_result worker(void)
+ close(STDIN_FILENO);
+ close(STDOUT_FILENO);
+
+- /* Only WR_OK should result in a non-zero exit code */
+- return wr != WR_OK;
++ /* Only WR_OK and WR_HANGUP should result in a non-zero exit code */
++ return wr != WR_OK && wr != WR_HANGUP;
+ }
+
+ static int max_connections = 32;
6: 252098db219 ! 6: 43f1cdcbb82 test-http-server: add HTTP request parsing
@@ Commit message
test-http-server: add HTTP request parsing
Add ability to parse HTTP requests to the test-http-server test helper.
+ Introduce `struct req` to store request information including:
+
+ * HTTP method & version
+ * Request path and query parameters
+ * Headers
+ * Content type and length (from `Content-Type` and `-Length` headers)
+
+ Failure to parse the request results in a 400 Bad Request response to
+ the client. Note that we're not trying to support all possible requests
+ here, but just enough to exercise all code under test.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
## t/helper/test-http-server.c ##
@@ t/helper/test-http-server.c: enum worker_result {
- WR_HANGUP = 1<<1,
- };
-
+ * Close the socket and clean up. Does not imply an error.
+ */
+ WR_HANGUP = 2,
++
++ /*
++ * Unexpected request message or error in request parsing.
++ * Respond with an 400 error. Close the socket and cleanup.
++ * Exit child-process with a non-zero status.
++ */
++ WR_CLIENT_ERROR = 3,
++};
++
+/*
+ * Fields from a parsed HTTP request.
+ */
@@ t/helper/test-http-server.c: enum worker_result {
+
+ struct string_list header_list;
+ const char *content_type;
-+ ssize_t content_length;
-+};
-+
++ uintmax_t content_length;
++ unsigned has_content_length:1;
+ };
+
+#define REQ__INIT { \
+ .start_line = STRBUF_INIT, \
+ .uri_path = STRBUF_INIT, \
+ .query_args = STRBUF_INIT, \
+ .header_list = STRING_LIST_INIT_NODUP, \
+ .content_type = NULL, \
-+ .content_length = -1 \
-+ }
++ .content_length = 0, \
++ .has_content_length = 0, \
++}
+
+static void req__release(struct req *req)
+{
@@ t/helper/test-http-server.c: enum worker_result {
+ string_list_clear(&req->header_list, 0);
+}
+
- static enum worker_result send_http_error(
- int fd,
- int http_code, const char *http_code_name,
+ static enum worker_result send_http_error(int fd, int http_code,
+ const char *http_code_name,
+ int retry_after_seconds,
@@ t/helper/test-http-server.c: done:
return wr;
}
@@ t/helper/test-http-server.c: done:
+ *
+ */
+ if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
-+ result = WR_OK | WR_HANGUP;
++ result = WR_HANGUP;
+ goto done;
+ }
+
@@ t/helper/test-http-server.c: done:
+ if (nr_start_line_fields != 3) {
+ logerror("could not parse request start-line '%s'",
+ req->start_line.buf);
-+ result = WR_IO_ERROR;
++ result = WR_CLIENT_ERROR;
+ goto done;
+ }
+
@@ t/helper/test-http-server.c: done:
+ if (strcmp(req->http_version, "HTTP/1.1")) {
+ logerror("unsupported version '%s' (expecting HTTP/1.1)",
+ req->http_version);
-+ result = WR_IO_ERROR;
++ result = WR_CLIENT_ERROR;
+ goto done;
+ }
+
@@ t/helper/test-http-server.c: done:
+ string_list_append(&req->header_list, hp);
+
+ /* also store common request headers as struct req members */
-+ if (skip_prefix(hp, "Content-Type: ", &hv)) {
++ if (skip_iprefix(hp, "Content-Type: ", &hv)) {
+ req->content_type = hv;
-+ } else if (skip_prefix(hp, "Content-Length: ", &hv)) {
-+ req->content_length = strtol(hv, &hp, 10);
++ } else if (skip_iprefix(hp, "Content-Length: ", &hv)) {
++ /*
++ * Content-Length is always non-negative, but has no
++ * upper bound according to RFC 7230 (§3.3.2).
++ */
++ intmax_t len = 0;
++ if (sscanf(hv, "%"PRIdMAX, &len) != 1 || len < 0 ||
++ len == INTMAX_MAX) {
++ logerror("invalid content-length: '%s'", hv);
++ result = WR_CLIENT_ERROR;
++ goto done;
++ }
++
++ req->content_length = (uintmax_t)len;
++ req->has_content_length = 1;
+ }
+ }
+
@@ t/helper/test-http-server.c: done:
+ trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
+ trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
+ trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
-+ if (req->content_length >= 0)
-+ trace2_printf("%s: clen: %d", TR2_CAT, req->content_length);
++ if (req->has_content_length)
++ trace2_printf("%s: clen: %"PRIuMAX, TR2_CAT,
++ req->content_length);
+ if (req->content_type)
+ trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
+ for_each_string_list_item(item, &req->header_list)
@@ t/helper/test-http-server.c: done:
+static enum worker_result dispatch(struct req *req)
+{
+ return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
-+ WR_OK | WR_HANGUP);
++ WR_HANGUP);
+}
+
static enum worker_result worker(void)
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
while (1) {
- wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
-- NULL, WR_OK | WR_HANGUP);
+- NULL, WR_HANGUP);
+ req__release(&req);
+
+ alarm(timeout);
+ wr = req__read(&req, 0);
+ alarm(0);
+
++ if (wr == WR_CLIENT_ERROR)
++ wr = send_http_error(STDOUT_FILENO, 400, "Bad Request",
++ -1, NULL, wr);
++
+ if (wr != WR_OK)
+ break;
@@ t/helper/test-http-server.c: static enum worker_result worker(void)
if (wr != WR_OK)
break;
}
+
+ ## t/t5556-http-auth.sh (new) ##
+@@
++#!/bin/sh
++
++test_description='test http auth header and credential helper interop'
++
++TEST_NO_CREATE_REPO=1
++. ./test-lib.sh
++
++# Setup a repository
++#
++REPO_DIR="$TRASH_DIRECTORY"/repo
++
++SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
++
++PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
++
++test_expect_success 'setup repos' '
++ test_create_repo "$REPO_DIR" &&
++ git -C "$REPO_DIR" branch -M main
++'
++
++run_http_server_worker() {
++ (
++ cd "$REPO_DIR"
++ test-http-server --worker "$@" 2>"$SERVER_LOG" | tr -d "\r"
++ )
++}
++
++per_test_cleanup () {
++ rm -f OUT.* &&
++ rm -f IN.* &&
++}
++
++test_expect_success 'http auth server request parsing' '
++ test_when_finished "per_test_cleanup" &&
++
++ cat >auth.config <<-EOF &&
++ [auth]
++ allowAnonymous = true
++ EOF
++
++ echo "HTTP/1.1 400 Bad Request" >OUT.http400 &&
++ echo "HTTP/1.1 200 OK" >OUT.http200 &&
++
++ cat >IN.http.valid <<-EOF &&
++ GET /info/refs HTTP/1.1
++ Content-Length: 0
++ EOF
++
++ cat >IN.http.badfirstline <<-EOF &&
++ /info/refs GET HTTP
++ EOF
++
++ cat >IN.http.badhttpver <<-EOF &&
++ GET /info/refs HTTP/999.9
++ EOF
++
++ cat >IN.http.ltzlen <<-EOF &&
++ GET /info/refs HTTP/1.1
++ Content-Length: -1
++ EOF
++
++ cat >IN.http.badlen <<-EOF &&
++ GET /info/refs HTTP/1.1
++ Content-Length: not-a-number
++ EOF
++
++ cat >IN.http.overlen <<-EOF &&
++ GET /info/refs HTTP/1.1
++ Content-Length: 9223372036854775807
++ EOF
++
++ run_http_server_worker \
++ --auth-config="$TRASH_DIRECTORY/auth.config" <IN.http.valid \
++ | head -n1 >OUT.actual &&
++ test_cmp OUT.http200 OUT.actual &&
++
++ run_http_server_worker <IN.http.badfirstline | head -n1 >OUT.actual &&
++ test_cmp OUT.http400 OUT.actual &&
++
++ run_http_server_worker <IN.http.ltzlen | head -n1 >OUT.actual &&
++ test_cmp OUT.http400 OUT.actual &&
++
++ run_http_server_worker <IN.http.badlen | head -n1 >OUT.actual &&
++ test_cmp OUT.http400 OUT.actual &&
++
++ run_http_server_worker <IN.http.overlen | head -n1 >OUT.actual &&
++ test_cmp OUT.http400 OUT.actual
++'
++
++test_done
7: ab06ac9b965 ! 7: ca9c2787248 test-http-server: pass Git requests to http-backend
@@ t/helper/test-http-server.c: done:
+ return error(_("could not send '%s'"), ok);
+
+ strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
-+ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
-+ req->uri_path.buf);
++ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
+ strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
+ if (req->query_args.len)
-+ strvec_pushf(&cp.env, "QUERY_STRING=%s",
-+ req->query_args.buf);
++ strvec_pushf(&cp.env, "QUERY_STRING=%s", req->query_args.buf);
+ if (req->content_type)
-+ strvec_pushf(&cp.env, "CONTENT_TYPE=%s",
-+ req->content_type);
-+ if (req->content_length >= 0)
-+ strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIdMAX,
-+ (intmax_t)req->content_length);
++ strvec_pushf(&cp.env, "CONTENT_TYPE=%s", req->content_type);
++ if (req->has_content_length)
++ strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIuMAX,
++ (uintmax_t)req->content_length);
+ cp.git_cmd = 1;
+ strvec_push(&cp.args, "http-backend");
+ res = run_command(&cp);
@@ t/helper/test-http-server.c: done:
+ return do__git(req);
+
return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
- WR_OK | WR_HANGUP);
+ WR_HANGUP);
}
- ## t/t5556-http-auth.sh (new) ##
-@@
-+#!/bin/sh
-+
-+test_description='test http auth header and credential helper interop'
-+
-+TEST_NO_CREATE_REPO=1
-+. ./test-lib.sh
-+
+ ## t/t5556-http-auth.sh ##
+@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'
+ TEST_NO_CREATE_REPO=1
+ . ./test-lib.sh
+
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
-+# Setup a repository
-+#
-+REPO_DIR="$TRASH_DIRECTORY"/repo
-+
+ # Setup a repository
+ #
+ REPO_DIR="$TRASH_DIRECTORY"/repo
+
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
+# any need to configure directory mappings etc - we only serve this
@@ t/t5556-http-auth.sh (new)
+# killing it by PID).
+#
+PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
-+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
-+
-+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
-+
-+test_expect_success 'setup repos' '
-+ test_create_repo "$REPO_DIR" &&
-+ git -C "$REPO_DIR" branch -M main
-+'
-+
+ SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
+
+ PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+@@ t/t5556-http-auth.sh: run_http_server_worker() {
+ )
+ }
+
+stop_http_server () {
+ if ! test -f "$PID_FILE"
+ then
@@ t/t5556-http-auth.sh (new)
+ return 1
+}
+
-+per_test_cleanup () {
+ per_test_cleanup () {
+ stop_http_server &&
-+ rm -f OUT.*
-+}
+ rm -f OUT.* &&
+ rm -f IN.* &&
+ }
+@@ t/t5556-http-auth.sh: test_expect_success 'http auth server request parsing' '
+ test_cmp OUT.http400 OUT.actual
+ '
+
+
+test_expect_success 'http auth anonymous no challenge' '
+ test_when_finished "per_test_cleanup" &&
@@ t/t5556-http-auth.sh (new)
+ git ls-remote $ORIGIN_URL
+'
+
-+test_done
+ test_done
8: a1ff55dd6e2 ! 8: b8d3e81b553 test-http-server: add simple authentication
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
+
strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
- strvec_pushf(&cp.env, "PATH_TRANSLATED=%s",
- req->uri_path.buf);
+ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
+ strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
return !!res;
}
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+
+static struct auth_module *get_auth_module(const char *scheme, int create)
+{
-+ int i;
+ struct auth_module *mod;
-+ for (i = 0; i < auth_modules_nr; i++) {
++ for (size_t i = 0; i < auth_modules_nr; i++) {
+ mod = auth_modules[i];
+ if (!strcasecmp(mod->scheme, scheme))
+ return mod;
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ struct auth_module *mod = xmalloc(sizeof(struct auth_module));
+ mod->scheme = xstrdup(scheme);
+ mod->challenge_params = NULL;
-+ CALLOC_ARRAY(mod->tokens, 1);
++ ALLOC_ARRAY(mod->tokens, 1);
+ string_list_init_dup(mod->tokens);
+
+ ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ for_each_string_list_item(hdr, &req->header_list) {
+ if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
+ split = strbuf_split_str(v, ' ', 2);
-+ if (!split[0] || !split[1]) continue;
-+
-+ /* trim trailing space ' ' */
-+ strbuf_setlen(split[0], split[0]->len - 1);
-+
-+ mod = get_auth_module(split[0]->buf, 0);
-+ if (mod) {
-+ result = AUTH_DENY;
-+
-+ for_each_string_list_item(token, mod->tokens) {
-+ if (!strcmp(split[1]->buf, token->string)) {
-+ result = AUTH_ALLOW;
-+ break;
++ if (split[0] && split[1]) {
++ /* trim trailing space ' ' */
++ strbuf_rtrim(split[0]);
++
++ mod = get_auth_module(split[0]->buf, 0);
++ if (mod) {
++ result = AUTH_DENY;
++
++ for_each_string_list_item(token, mod->tokens) {
++ if (!strcmp(split[1]->buf, token->string)) {
++ result = AUTH_ALLOW;
++ break;
++ }
+ }
-+ }
+
-+ goto done;
++ strbuf_list_free(split);
++ goto done;
++ }
+ }
++
++ strbuf_list_free(split);
+ }
+ }
+
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ &hdrs, *wr);
+ }
+
-+ strbuf_list_free(split);
+ string_list_clear(&hdrs, 0);
+
+ return result == AUTH_ALLOW ||
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ return -1;
+
+ /* trim trailing ':' */
-+ if (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')
++ if (p[0]->len && p[0]->buf[p[0]->len - 1] == ':')
+ strbuf_setlen(p[0], p[0]->len - 1);
+
+ *scheme = strbuf_detach(p[0], NULL);
-+
-+ if (p[1])
-+ *val = strbuf_detach(p[1], NULL);
++ *val = p[1] ? strbuf_detach(p[1], NULL) : NULL;
+
+ strbuf_list_free(p);
+ return 0;
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ char *scheme = NULL;
+ char *token = NULL;
+ char *challenge = NULL;
-+ struct auth_module *mod = NULL;
++ struct auth_module *mod;
+
+ if (!strcmp(name, "auth.challenge")) {
+ if (split_auth_param(val, &scheme, &challenge)) {
@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)
+ return do__git(req, user);
return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
- WR_OK | WR_HANGUP);
+ WR_HANGUP);
@@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
pid_file = v;
continue;
@@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)
## t/t5556-http-auth.sh ##
@@ t/t5556-http-auth.sh: per_test_cleanup () {
- rm -f OUT.*
+ stop_http_server &&
+ rm -f OUT.* &&
+ rm -f IN.* &&
++ rm -f auth.config
}
+ test_expect_success 'http auth server request parsing' '
+@@ t/t5556-http-auth.sh: test_expect_success 'http auth server request parsing' '
+ test_cmp OUT.http400 OUT.actual
+ '
+
+test_expect_success CURL 'http auth server auth config' '
-+ #test_when_finished "per_test_cleanup" &&
++ test_when_finished "per_test_cleanup" &&
+
+ cat >auth.config <<-EOF &&
+ [auth]
@@ t/t5556-http-auth.sh: per_test_cleanup () {
+
+ test_cmp OUT.expected OUT.actual
+'
-+
+
test_expect_success 'http auth anonymous no challenge' '
test_when_finished "per_test_cleanup" &&
9: 76125cdf239 = 9: 2f97c94f679 test-http-server: add sending of arbitrary headers
10: cc9a220ed1f = 10: 4b1635b3f69 http: replace unsafe size_t multiplication with st_mult
11: bc1ac8d3eb3 = 11: 5f5e46038cf http: read HTTP WWW-Authenticate response headers
12: 7c8229f0b11 ! 12: 09164f77d56 credential: add WWW-Authenticate header to cred requests
@@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '
+setup_credential_helper
+
- stop_http_server () {
- if ! test -f "$PID_FILE"
- then
-@@ t/t5556-http-auth.sh: start_http_server () {
-
- per_test_cleanup () {
+ run_http_server_worker() {
+ (
+ cd "$REPO_DIR"
+@@ t/t5556-http-auth.sh: per_test_cleanup () {
stop_http_server &&
-- rm -f OUT.*
-+ rm -f OUT.* &&
+ rm -f OUT.* &&
+ rm -f IN.* &&
+ rm -f *.cred &&
-+ rm -f auth.config
+ rm -f auth.config
}
- test_expect_success CURL 'http auth server auth config' '
@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '
git ls-remote $ORIGIN_URL
'
--
gitgitgadget
^ permalink raw reply
* [PATCH v7 05/12] test-http-server: add HTTP error response function
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Introduce a function to the test-http-server test helper to write more
full and valid HTTP error responses, including all the standard response
headers like `Server` and `Date`.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
t/helper/test-http-server.c | 76 +++++++++++++++++++++++++++++++++----
1 file changed, 69 insertions(+), 7 deletions(-)
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 6e9a1c479ce..7ca4ddc7999 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -76,11 +76,75 @@ enum worker_result {
* Exit child-process with non-zero status.
*/
WR_FATAL_ERROR = 1,
+
+ /*
+ * Close the socket and clean up. Does not imply an error.
+ */
+ WR_HANGUP = 2,
};
+static enum worker_result send_http_error(int fd, int http_code,
+ const char *http_code_name,
+ int retry_after_seconds,
+ struct string_list *response_headers,
+ enum worker_result wr_in)
+{
+ struct strbuf response_header = STRBUF_INIT;
+ struct strbuf response_content = STRBUF_INIT;
+ struct string_list_item *h;
+ enum worker_result wr;
+
+ strbuf_addf(&response_content, "Error: %d %s\r\n", http_code,
+ http_code_name);
+
+ if (retry_after_seconds > 0)
+ strbuf_addf(&response_content, "Retry-After: %d\r\n",
+ retry_after_seconds);
+
+ strbuf_addf(&response_header, "HTTP/1.1 %d %s\r\n", http_code,
+ http_code_name);
+ strbuf_addstr(&response_header, "Cache-Control: private\r\n");
+ strbuf_addstr(&response_header, "Content-Type: text/plain\r\n");
+ strbuf_addf(&response_header, "Content-Length: %"PRIuMAX"\r\n",
+ (uintmax_t)response_content.len);
+
+ if (retry_after_seconds > 0)
+ strbuf_addf(&response_header, "Retry-After: %d\r\n",
+ retry_after_seconds);
+
+ strbuf_addf(&response_header, "Server: test-http-server/%s\r\n",
+ git_version_string);
+ strbuf_addf(&response_header, "Date: %s\r\n", show_date(time(NULL), 0,
+ DATE_MODE(RFC2822)));
+
+ if (response_headers)
+ for_each_string_list_item(h, response_headers)
+ strbuf_addf(&response_header, "%s\r\n", h->string);
+ strbuf_addstr(&response_header, "\r\n");
+
+ if (write_in_full(fd, response_header.buf, response_header.len) < 0) {
+ logerror("unable to write response header");
+ wr = WR_FATAL_ERROR;
+ goto done;
+ }
+
+ if (write_in_full(fd, response_content.buf, response_content.len) < 0) {
+ logerror("unable to write response content body");
+ wr = WR_FATAL_ERROR;
+ goto done;
+ }
+
+ wr = wr_in;
+
+done:
+ strbuf_release(&response_header);
+ strbuf_release(&response_content);
+
+ return wr;
+}
+
static enum worker_result worker(void)
{
- const char *response = "HTTP/1.1 501 Not Implemented\r\n";
char *client_addr = getenv("REMOTE_ADDR");
char *client_port = getenv("REMOTE_PORT");
enum worker_result wr = WR_OK;
@@ -91,10 +155,8 @@ static enum worker_result worker(void)
set_keep_alive(0, logerror);
while (1) {
- if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
- logerror("unable to write response");
- wr = WR_FATAL_ERROR;
- }
+ wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
+ NULL, WR_HANGUP);
if (wr != WR_OK)
break;
@@ -103,8 +165,8 @@ static enum worker_result worker(void)
close(STDIN_FILENO);
close(STDOUT_FILENO);
- /* Only WR_OK should result in a non-zero exit code */
- return wr != WR_OK;
+ /* Only WR_OK and WR_HANGUP should result in a non-zero exit code */
+ return wr != WR_OK && wr != WR_HANGUP;
}
static int max_connections = 32;
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 02/12] daemon: libify child process handling functions
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Extract functions and structures for managing child processes started
from the parent daemon-like process from `daemon.c` to the new shared
`daemon-utils.{c,h}` files.
One minor functional change is introduced to `check_dead_children()`
where the logging of a dead/disconnected child is now optional. With the
'libification' of these functions we extract the call to `loginfo` to a
call to a function pointer, and guard the log message creation and
logging behind a `NULL` check. Callers can now skip logging by passing
`NULL` as the `log_fn loginfo` argument.
The behaviour of callers in `daemon.c` remains the same (save one extra
NULL check) however as a pointer to `loginfo` is always passed.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++
daemon-utils.h | 32 ++++++++++++++++++
daemon.c | 92 +++-----------------------------------------------
3 files changed, 114 insertions(+), 87 deletions(-)
diff --git a/daemon-utils.c b/daemon-utils.c
index b96b55962db..8506664b440 100644
--- a/daemon-utils.c
+++ b/daemon-utils.c
@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port,
}
}
}
+
+static int addrcmp(const struct sockaddr_storage *s1,
+ const struct sockaddr_storage *s2)
+{
+ const struct sockaddr *sa1 = (const struct sockaddr*) s1;
+ const struct sockaddr *sa2 = (const struct sockaddr*) s2;
+
+ if (sa1->sa_family != sa2->sa_family)
+ return sa1->sa_family - sa2->sa_family;
+ if (sa1->sa_family == AF_INET)
+ return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
+ &((struct sockaddr_in *)s2)->sin_addr,
+ sizeof(struct in_addr));
+#ifndef NO_IPV6
+ if (sa1->sa_family == AF_INET6)
+ return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
+ &((struct sockaddr_in6 *)s2)->sin6_addr,
+ sizeof(struct in6_addr));
+#endif
+ return 0;
+}
+
+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
+ struct child *firstborn , unsigned int *live_children)
+{
+ struct child *newborn, **cradle;
+
+ CALLOC_ARRAY(newborn, 1);
+ (*live_children)++;
+ memcpy(&newborn->cld, cld, sizeof(*cld));
+ memcpy(&newborn->address, addr, addrlen);
+ for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
+ if (!addrcmp(&(*cradle)->address, &newborn->address))
+ break;
+ newborn->next = *cradle;
+ *cradle = newborn;
+}
+
+void kill_some_child(struct child *firstborn)
+{
+ const struct child *blanket, *next;
+
+ if (!(blanket = firstborn))
+ return;
+
+ for (; (next = blanket->next); blanket = next)
+ if (!addrcmp(&blanket->address, &next->address)) {
+ kill(blanket->cld.pid, SIGTERM);
+ break;
+ }
+}
+
+void check_dead_children(struct child *firstborn, unsigned int *live_children,
+ log_fn loginfo)
+{
+ int status;
+ pid_t pid;
+
+ struct child **cradle, *blanket;
+ for (cradle = &firstborn; (blanket = *cradle);)
+ if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+ if (loginfo) {
+ const char *dead = "";
+ if (status)
+ dead = " (with error)";
+ loginfo("[%"PRIuMAX"] Disconnected%s",
+ (uintmax_t)pid, dead);
+ }
+
+ /* remove the child */
+ *cradle = blanket->next;
+ (*live_children)--;
+ child_process_clear(&blanket->cld);
+ free(blanket);
+ } else
+ cradle = &blanket->next;
+}
diff --git a/daemon-utils.h b/daemon-utils.h
index 6710a2a6dc0..97e5cae20b8 100644
--- a/daemon-utils.h
+++ b/daemon-utils.h
@@ -2,6 +2,7 @@
#define DAEMON_UTILS_H
#include "git-compat-util.h"
+#include "run-command.h"
#include "string-list.h"
typedef void (*log_fn)(const char *msg, ...);
@@ -20,4 +21,35 @@ void socksetup(struct string_list *listen_addr, int listen_port,
struct socketlist *socklist, int reuseaddr,
log_fn logerror);
+struct child {
+ struct child *next;
+ struct child_process cld;
+ struct sockaddr_storage address;
+};
+
+/*
+ * Add the child_process to the set of children and increment the number of
+ * live children.
+ */
+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
+ struct child *firstborn, unsigned int *live_children);
+
+/*
+ * Kill the newest connection from a duplicate IP.
+ *
+ * This function should be called if the number of connections grows
+ * past the maximum number of allowed connections.
+ */
+void kill_some_child(struct child *firstborn);
+
+/*
+ * Check for children that have disconnected and remove them from the
+ * active set, decrementing the number of live children.
+ *
+ * Optionally log the child PID that disconnected by passing a loginfo
+ * function.
+ */
+void check_dead_children(struct child *firstborn, unsigned int *live_children,
+ log_fn loginfo);
+
#endif
diff --git a/daemon.c b/daemon.c
index 1ed4e705680..ec3b407ecbc 100644
--- a/daemon.c
+++ b/daemon.c
@@ -785,93 +785,11 @@ static int execute(void)
return -1;
}
-static int addrcmp(const struct sockaddr_storage *s1,
- const struct sockaddr_storage *s2)
-{
- const struct sockaddr *sa1 = (const struct sockaddr*) s1;
- const struct sockaddr *sa2 = (const struct sockaddr*) s2;
-
- if (sa1->sa_family != sa2->sa_family)
- return sa1->sa_family - sa2->sa_family;
- if (sa1->sa_family == AF_INET)
- return memcmp(&((struct sockaddr_in *)s1)->sin_addr,
- &((struct sockaddr_in *)s2)->sin_addr,
- sizeof(struct in_addr));
-#ifndef NO_IPV6
- if (sa1->sa_family == AF_INET6)
- return memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,
- &((struct sockaddr_in6 *)s2)->sin6_addr,
- sizeof(struct in6_addr));
-#endif
- return 0;
-}
-
static int max_connections = 32;
static unsigned int live_children;
-static struct child {
- struct child *next;
- struct child_process cld;
- struct sockaddr_storage address;
-} *firstborn;
-
-static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)
-{
- struct child *newborn, **cradle;
-
- CALLOC_ARRAY(newborn, 1);
- live_children++;
- memcpy(&newborn->cld, cld, sizeof(*cld));
- memcpy(&newborn->address, addr, addrlen);
- for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
- if (!addrcmp(&(*cradle)->address, &newborn->address))
- break;
- newborn->next = *cradle;
- *cradle = newborn;
-}
-
-/*
- * This gets called if the number of connections grows
- * past "max_connections".
- *
- * We kill the newest connection from a duplicate IP.
- */
-static void kill_some_child(void)
-{
- const struct child *blanket, *next;
-
- if (!(blanket = firstborn))
- return;
-
- for (; (next = blanket->next); blanket = next)
- if (!addrcmp(&blanket->address, &next->address)) {
- kill(blanket->cld.pid, SIGTERM);
- break;
- }
-}
-
-static void check_dead_children(void)
-{
- int status;
- pid_t pid;
-
- struct child **cradle, *blanket;
- for (cradle = &firstborn; (blanket = *cradle);)
- if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
- const char *dead = "";
- if (status)
- dead = " (with error)";
- loginfo("[%"PRIuMAX"] Disconnected%s", (uintmax_t)pid, dead);
-
- /* remove the child */
- *cradle = blanket->next;
- live_children--;
- child_process_clear(&blanket->cld);
- free(blanket);
- } else
- cradle = &blanket->next;
-}
+static struct child *firstborn;
static struct strvec cld_argv = STRVEC_INIT;
static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
@@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
struct child_process cld = CHILD_PROCESS_INIT;
if (max_connections && live_children >= max_connections) {
- kill_some_child();
+ kill_some_child(firstborn);
sleep(1); /* give it some time to die */
- check_dead_children();
+ check_dead_children(firstborn, &live_children, loginfo);
if (live_children >= max_connections) {
close(incoming);
logerror("Too many children, dropping connection");
@@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
if (start_command(&cld))
logerror("unable to fork");
else
- add_child(&cld, addr, addrlen);
+ add_child(&cld, addr, addrlen, firstborn, &live_children);
}
static void child_handler(int signo)
@@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
for (;;) {
int i;
- check_dead_children();
+ check_dead_children(firstborn, &live_children, loginfo);
if (poll(pfd, socklist->nr, -1) < 0) {
if (errno != EINTR) {
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 03/12] daemon: rename some esoteric/laboured terminology
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Rename some of the variables and function arguments used to manage child
processes. The existing names are esoteric; stretching an analogy too
far to the point of being confusing to understand.
Rename "firstborn" to "first_child", "newborn" to "new_cld", "blanket"
to "current" and "cradle" to "ptr".
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
daemon-utils.c | 46 +++++++++++++++++++++++-----------------------
daemon-utils.h | 6 +++---
daemon.c | 10 +++++-----
3 files changed, 31 insertions(+), 31 deletions(-)
diff --git a/daemon-utils.c b/daemon-utils.c
index 8506664b440..f23ea35ed7b 100644
--- a/daemon-utils.c
+++ b/daemon-utils.c
@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1,
}
void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
- struct child *firstborn , unsigned int *live_children)
+ struct child *first_child, unsigned int *live_children)
{
- struct child *newborn, **cradle;
+ struct child *new_cld, **current;
- CALLOC_ARRAY(newborn, 1);
+ CALLOC_ARRAY(new_cld, 1);
(*live_children)++;
- memcpy(&newborn->cld, cld, sizeof(*cld));
- memcpy(&newborn->address, addr, addrlen);
- for (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)
- if (!addrcmp(&(*cradle)->address, &newborn->address))
+ memcpy(&new_cld->cld, cld, sizeof(*cld));
+ memcpy(&new_cld->address, addr, addrlen);
+ for (current = &first_child; *current; current = &(*current)->next)
+ if (!addrcmp(&(*current)->address, &new_cld->address))
break;
- newborn->next = *cradle;
- *cradle = newborn;
+ new_cld->next = *current;
+ *current = new_cld;
}
-void kill_some_child(struct child *firstborn)
+void kill_some_child(struct child *first_child)
{
- const struct child *blanket, *next;
+ const struct child *current, *next;
- if (!(blanket = firstborn))
+ if (!(current = first_child))
return;
- for (; (next = blanket->next); blanket = next)
- if (!addrcmp(&blanket->address, &next->address)) {
- kill(blanket->cld.pid, SIGTERM);
+ for (; (next = current->next); current = next)
+ if (!addrcmp(¤t->address, &next->address)) {
+ kill(current->cld.pid, SIGTERM);
break;
}
}
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
+void check_dead_children(struct child *first_child, unsigned int *live_children,
log_fn loginfo)
{
int status;
pid_t pid;
- struct child **cradle, *blanket;
- for (cradle = &firstborn; (blanket = *cradle);)
- if ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {
+ struct child **ptr, *current;
+ for (ptr = &first_child; (current = *ptr);)
+ if ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {
if (loginfo) {
const char *dead = "";
if (status)
@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children,
}
/* remove the child */
- *cradle = blanket->next;
+ *ptr = current->next;
(*live_children)--;
- child_process_clear(&blanket->cld);
- free(blanket);
+ child_process_clear(¤t->cld);
+ free(current);
} else
- cradle = &blanket->next;
+ ptr = ¤t->next;
}
diff --git a/daemon-utils.h b/daemon-utils.h
index 97e5cae20b8..c866e9c9a4e 100644
--- a/daemon-utils.h
+++ b/daemon-utils.h
@@ -32,7 +32,7 @@ struct child {
* live children.
*/
void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,
- struct child *firstborn, unsigned int *live_children);
+ struct child *first_child, unsigned int *live_children);
/*
* Kill the newest connection from a duplicate IP.
@@ -40,7 +40,7 @@ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl
* This function should be called if the number of connections grows
* past the maximum number of allowed connections.
*/
-void kill_some_child(struct child *firstborn);
+void kill_some_child(struct child *first_child);
/*
* Check for children that have disconnected and remove them from the
@@ -49,7 +49,7 @@ void kill_some_child(struct child *firstborn);
* Optionally log the child PID that disconnected by passing a loginfo
* function.
*/
-void check_dead_children(struct child *firstborn, unsigned int *live_children,
+void check_dead_children(struct child *first_child, unsigned int *live_children,
log_fn loginfo);
#endif
diff --git a/daemon.c b/daemon.c
index ec3b407ecbc..d3e7d81de18 100644
--- a/daemon.c
+++ b/daemon.c
@@ -789,7 +789,7 @@ static int max_connections = 32;
static unsigned int live_children;
-static struct child *firstborn;
+static struct child *first_child;
static struct strvec cld_argv = STRVEC_INIT;
static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
struct child_process cld = CHILD_PROCESS_INIT;
if (max_connections && live_children >= max_connections) {
- kill_some_child(firstborn);
+ kill_some_child(first_child);
sleep(1); /* give it some time to die */
- check_dead_children(firstborn, &live_children, loginfo);
+ check_dead_children(first_child, &live_children, loginfo);
if (live_children >= max_connections) {
close(incoming);
logerror("Too many children, dropping connection");
@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
if (start_command(&cld))
logerror("unable to fork");
else
- add_child(&cld, addr, addrlen, firstborn, &live_children);
+ add_child(&cld, addr, addrlen, first_child, &live_children);
}
static void child_handler(int signo)
@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)
for (;;) {
int i;
- check_dead_children(firstborn, &live_children, loginfo);
+ check_dead_children(first_child, &live_children, loginfo);
if (poll(pfd, socklist->nr, -1) < 0) {
if (errno != EINTR) {
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 07/12] test-http-server: pass Git requests to http-backend
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Teach the test-http-sever test helper to forward Git requests to the
`git-http-backend`.
Introduce a new test script t5556-http-auth.sh that spins up the test
HTTP server and attempts an `ls-remote` on the served repository,
without any authentication.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
t/helper/test-http-server.c | 68 ++++++++++++++++++++++++++++++
t/t5556-http-auth.sh | 83 +++++++++++++++++++++++++++++++++++++
2 files changed, 151 insertions(+)
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 900f5733cc1..4191daf3c64 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -323,8 +323,76 @@ done:
return result;
}
+static int is_git_request(struct req *req)
+{
+ static regex_t *smart_http_regex;
+ static int initialized;
+
+ if (!initialized) {
+ smart_http_regex = xmalloc(sizeof(*smart_http_regex));
+ /*
+ * This regular expression matches all dumb and smart HTTP
+ * requests that are currently in use, and defined in
+ * Documentation/gitprotocol-http.txt.
+ *
+ */
+ if (regcomp(smart_http_regex, "^/(HEAD|info/refs|"
+ "objects/info/[^/]+|git-(upload|receive)-pack)$",
+ REG_EXTENDED)) {
+ warning("could not compile smart HTTP regex");
+ smart_http_regex = NULL;
+ }
+ initialized = 1;
+ }
+
+ return smart_http_regex &&
+ !regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
+}
+
+static enum worker_result do__git(struct req *req)
+{
+ const char *ok = "HTTP/1.1 200 OK\r\n";
+ struct child_process cp = CHILD_PROCESS_INIT;
+ int res;
+
+ /*
+ * Note that we always respond with a 200 OK response even if the
+ * http-backend process exits with an error. This helper is intended
+ * only to be used to exercise the HTTP auth handling in the Git client,
+ * and specifically around authentication (not handled by http-backend).
+ *
+ * If we wanted to respond with a more 'valid' HTTP response status then
+ * we'd need to buffer the output of http-backend, wait for and grok the
+ * exit status of the process, then write the HTTP status line followed
+ * by the http-backend output. This is outside of the scope of this test
+ * helper's use at time of writing.
+ */
+ if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
+ return error(_("could not send '%s'"), ok);
+
+ strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
+ strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
+ strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
+ if (req->query_args.len)
+ strvec_pushf(&cp.env, "QUERY_STRING=%s", req->query_args.buf);
+ if (req->content_type)
+ strvec_pushf(&cp.env, "CONTENT_TYPE=%s", req->content_type);
+ if (req->has_content_length)
+ strvec_pushf(&cp.env, "CONTENT_LENGTH=%" PRIuMAX,
+ (uintmax_t)req->content_length);
+ cp.git_cmd = 1;
+ strvec_push(&cp.args, "http-backend");
+ res = run_command(&cp);
+ close(STDOUT_FILENO);
+ close(STDIN_FILENO);
+ return !!res;
+}
+
static enum worker_result dispatch(struct req *req)
{
+ if (is_git_request(req))
+ return do__git(req);
+
return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
WR_HANGUP);
}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index 06efc85ca53..c0a47ce342b 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -5,10 +5,25 @@ test_description='test http auth header and credential helper interop'
TEST_NO_CREATE_REPO=1
. ./test-lib.sh
+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT
+
# Setup a repository
#
REPO_DIR="$TRASH_DIRECTORY"/repo
+# Setup some lookback URLs where test-http-server will be listening.
+# We will spawn it directly inside the repo directory, so we avoid
+# any need to configure directory mappings etc - we only serve this
+# repository from the root '/' of the server.
+#
+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT
+ORIGIN_URL=http://$HOST_PORT/
+
+# The pid-file is created by test-http-server when it starts.
+# The server will shutdown if/when we delete it (this is easier than
+# killing it by PID).
+#
+PID_FILE="$TRASH_DIRECTORY"/pid-file.pid
SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
@@ -25,7 +40,65 @@ run_http_server_worker() {
)
}
+stop_http_server () {
+ if ! test -f "$PID_FILE"
+ then
+ return 0
+ fi
+ #
+ # The server will shutdown automatically when we delete the pid-file.
+ #
+ rm -f "$PID_FILE"
+ #
+ # Give it a few seconds to shutdown (mainly to completely release the
+ # port before the next test start another instance and it attempts to
+ # bind to it).
+ #
+ for k in 0 1 2 3 4
+ do
+ if grep -q "Starting graceful shutdown" "$SERVER_LOG"
+ then
+ return 0
+ fi
+ sleep 1
+ done
+
+ echo "stop_http_server: timeout waiting for server shutdown"
+ return 1
+}
+
+start_http_server () {
+ #
+ # Launch our server into the background in repo_dir.
+ #
+ (
+ cd "$REPO_DIR"
+ test-http-server --verbose \
+ --listen=127.0.0.1 \
+ --port=$GIT_TEST_HTTP_PROTOCOL_PORT \
+ --reuseaddr \
+ --pid-file="$PID_FILE" \
+ "$@" \
+ 2>"$SERVER_LOG" &
+ )
+ #
+ # Give it a few seconds to get started.
+ #
+ for k in 0 1 2 3 4
+ do
+ if test -f "$PID_FILE"
+ then
+ return 0
+ fi
+ sleep 1
+ done
+
+ echo "start_http_server: timeout waiting for server startup"
+ return 1
+}
+
per_test_cleanup () {
+ stop_http_server &&
rm -f OUT.* &&
rm -f IN.* &&
}
@@ -87,4 +160,14 @@ test_expect_success 'http auth server request parsing' '
test_cmp OUT.http400 OUT.actual
'
+
+test_expect_success 'http auth anonymous no challenge' '
+ test_when_finished "per_test_cleanup" &&
+
+ start_http_server &&
+
+ # Attempt to read from a protected repository
+ git ls-remote $ORIGIN_URL
+'
+
test_done
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 04/12] test-http-server: add stub HTTP server test helper
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Introduce a mini HTTP server helper that in the future will be enhanced
to provide a frontend for the git-http-backend, with support for
arbitrary authentication schemes.
Right now, test-http-server is a pared-down copy of the git-daemon that
always returns a 501 Not Implemented response to all callers.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
Makefile | 1 +
contrib/buildsystems/CMakeLists.txt | 11 +-
t/helper/.gitignore | 1 +
t/helper/test-http-server.c | 381 ++++++++++++++++++++++++++++
4 files changed, 392 insertions(+), 2 deletions(-)
create mode 100644 t/helper/test-http-server.c
diff --git a/Makefile b/Makefile
index 2654094dbb5..3cd61c792ac 100644
--- a/Makefile
+++ b/Makefile
@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o
# Do not add more tests here unless they have extra dependencies. Add
# them in TEST_BUILTINS_OBJS above.
TEST_PROGRAMS_NEED_X += test-fake-ssh
+TEST_PROGRAMS_NEED_X += test-http-server
TEST_PROGRAMS_NEED_X += test-tool
TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))
diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 2f6e0197ffa..5d949dcb16c 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -961,6 +961,9 @@ if(BUILD_TESTING)
add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)
target_link_libraries(test-fake-ssh common-main)
+add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)
+target_link_libraries(test-http-server common-main)
+
#reftable-tests
parse_makefile_for_sources(test-reftable_SOURCES "REFTABLE_TEST_OBJS")
list(TRANSFORM test-reftable_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/")
@@ -980,6 +983,11 @@ if(MSVC)
PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
set_target_properties(test-fake-ssh test-tool
PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
+
+ set_target_properties(test-http-server
+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)
+ set_target_properties(test-http-server
+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)
endif()
#wrapper scripts
@@ -987,8 +995,7 @@ set(wrapper_scripts
git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)
set(wrapper_test_scripts
- test-fake-ssh test-tool)
-
+ test-http-server test-fake-ssh test-tool)
foreach(script ${wrapper_scripts})
file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
diff --git a/t/helper/.gitignore b/t/helper/.gitignore
index 8c2ddcce95f..9aa9c752997 100644
--- a/t/helper/.gitignore
+++ b/t/helper/.gitignore
@@ -1,2 +1,3 @@
/test-tool
/test-fake-ssh
+/test-http-server
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
new file mode 100644
index 00000000000..6e9a1c479ce
--- /dev/null
+++ b/t/helper/test-http-server.c
@@ -0,0 +1,381 @@
+#include "daemon-utils.h"
+#include "config.h"
+#include "run-command.h"
+#include "strbuf.h"
+#include "string-list.h"
+#include "trace2.h"
+#include "version.h"
+#include "dir.h"
+#include "date.h"
+
+#define TR2_CAT "test-http-server"
+
+static const char *pid_file;
+static int verbose;
+static int reuseaddr;
+
+static const char test_http_auth_usage[] =
+"http-server [--verbose]\n"
+" [--timeout=<n>] [--max-connections=<n>]\n"
+" [--reuseaddr] [--pid-file=<file>]\n"
+" [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+;
+
+static unsigned int timeout;
+
+static void logreport(const char *label, const char *err, va_list params)
+{
+ struct strbuf msg = STRBUF_INIT;
+
+ strbuf_addf(&msg, "[%"PRIuMAX"] %s: ", (uintmax_t)getpid(), label);
+ strbuf_vaddf(&msg, err, params);
+ strbuf_addch(&msg, '\n');
+
+ fwrite(msg.buf, sizeof(char), msg.len, stderr);
+ fflush(stderr);
+
+ strbuf_release(&msg);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void logerror(const char *err, ...)
+{
+ va_list params;
+ va_start(params, err);
+ logreport("error", err, params);
+ va_end(params);
+}
+
+__attribute__((format (printf, 1, 2)))
+static void loginfo(const char *err, ...)
+{
+ va_list params;
+ if (!verbose)
+ return;
+ va_start(params, err);
+ logreport("info", err, params);
+ va_end(params);
+}
+
+/*
+ * The code in this section is used by "worker" instances to service
+ * a single connection from a client. The worker talks to the client
+ * on stdin and stdout.
+ */
+
+enum worker_result {
+ /*
+ * Operation successful.
+ * Caller *might* keep the socket open and allow keep-alive.
+ */
+ WR_OK = 0,
+
+ /*
+ * Fatal error that is not recoverable.
+ * Close the socket and clean up.
+ * Exit child-process with non-zero status.
+ */
+ WR_FATAL_ERROR = 1,
+};
+
+static enum worker_result worker(void)
+{
+ const char *response = "HTTP/1.1 501 Not Implemented\r\n";
+ char *client_addr = getenv("REMOTE_ADDR");
+ char *client_port = getenv("REMOTE_PORT");
+ enum worker_result wr = WR_OK;
+
+ if (client_addr)
+ loginfo("Connection from %s:%s", client_addr, client_port);
+
+ set_keep_alive(0, logerror);
+
+ while (1) {
+ if (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {
+ logerror("unable to write response");
+ wr = WR_FATAL_ERROR;
+ }
+
+ if (wr != WR_OK)
+ break;
+ }
+
+ close(STDIN_FILENO);
+ close(STDOUT_FILENO);
+
+ /* Only WR_OK should result in a non-zero exit code */
+ return wr != WR_OK;
+}
+
+static int max_connections = 32;
+
+static unsigned int live_children;
+
+static struct child *first_child;
+
+static struct strvec cld_argv = STRVEC_INIT;
+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)
+{
+ struct child_process cld = CHILD_PROCESS_INIT;
+
+ if (max_connections && live_children >= max_connections) {
+ kill_some_child(first_child);
+ sleep(1); /* give it some time to die */
+ check_dead_children(first_child, &live_children, loginfo);
+ if (live_children >= max_connections) {
+ close(incoming);
+ logerror("Too many children, dropping connection");
+ return;
+ }
+ }
+
+ if (addr->sa_family == AF_INET) {
+ char buf[128] = "";
+ struct sockaddr_in *sin_addr = (void *) addr;
+ inet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));
+ strvec_pushf(&cld.env, "REMOTE_ADDR=%s", buf);
+ strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+ ntohs(sin_addr->sin_port));
+#ifndef NO_IPV6
+ } else if (addr->sa_family == AF_INET6) {
+ char buf[128] = "";
+ struct sockaddr_in6 *sin6_addr = (void *) addr;
+ inet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));
+ strvec_pushf(&cld.env, "REMOTE_ADDR=[%s]", buf);
+ strvec_pushf(&cld.env, "REMOTE_PORT=%d",
+ ntohs(sin6_addr->sin6_port));
+#endif
+ }
+
+ strvec_pushv(&cld.args, cld_argv.v);
+ cld.in = incoming;
+ cld.out = dup(incoming);
+
+ if (cld.out < 0)
+ logerror("could not dup() `incoming`");
+ else if (start_command(&cld))
+ logerror("unable to fork");
+ else
+ add_child(&cld, addr, addrlen, first_child, &live_children);
+}
+
+static void child_handler(int signo)
+{
+ /*
+ * Otherwise empty handler because systemcalls will get interrupted
+ * upon signal receipt
+ * SysV needs the handler to be rearmed
+ */
+ signal(SIGCHLD, child_handler);
+}
+
+static int service_loop(struct socketlist *socklist)
+{
+ struct pollfd *pfd;
+ int i;
+
+ CALLOC_ARRAY(pfd, socklist->nr);
+
+ for (i = 0; i < socklist->nr; i++) {
+ pfd[i].fd = socklist->list[i];
+ pfd[i].events = POLLIN;
+ }
+
+ signal(SIGCHLD, child_handler);
+
+ for (;;) {
+ int i;
+ int nr_ready;
+ int timeout = (pid_file ? 100 : -1);
+
+ check_dead_children(first_child, &live_children, loginfo);
+
+ nr_ready = poll(pfd, socklist->nr, timeout);
+ if (nr_ready < 0) {
+ if (errno != EINTR) {
+ logerror("Poll failed, resuming: %s",
+ strerror(errno));
+ sleep(1);
+ }
+ continue;
+ }
+ else if (nr_ready == 0) {
+ /*
+ * If we have a pid_file, then we watch it.
+ * If someone deletes it, we shutdown the service.
+ * The shell scripts in the test suite will use this.
+ */
+ if (!pid_file || file_exists(pid_file))
+ continue;
+ goto shutdown;
+ }
+
+ for (i = 0; i < socklist->nr; i++) {
+ if (pfd[i].revents & POLLIN) {
+ union {
+ struct sockaddr sa;
+ struct sockaddr_in sai;
+#ifndef NO_IPV6
+ struct sockaddr_in6 sai6;
+#endif
+ } ss;
+ socklen_t sslen = sizeof(ss);
+ int incoming = accept(pfd[i].fd, &ss.sa, &sslen);
+ if (incoming < 0) {
+ switch (errno) {
+ case EAGAIN:
+ case EINTR:
+ case ECONNABORTED:
+ continue;
+ default:
+ die_errno("accept returned");
+ }
+ }
+ handle(incoming, &ss.sa, sslen);
+ }
+ }
+ }
+
+shutdown:
+ loginfo("Starting graceful shutdown (pid-file gone)");
+ for (i = 0; i < socklist->nr; i++)
+ close(socklist->list[i]);
+
+ return 0;
+}
+
+static int serve(struct string_list *listen_addr, int listen_port)
+{
+ struct socketlist socklist = { NULL, 0, 0 };
+
+ socksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);
+ if (socklist.nr == 0)
+ die("unable to allocate any listen sockets on port %u",
+ listen_port);
+
+ loginfo("Ready to rumble");
+
+ /*
+ * Wait to create the pid-file until we've setup the sockets
+ * and are open for business.
+ */
+ if (pid_file)
+ write_file(pid_file, "%"PRIuMAX, (uintmax_t) getpid());
+
+ return service_loop(&socklist);
+}
+
+/*
+ * This section is executed by both the primary instance and all
+ * worker instances. So, yes, each child-process re-parses the
+ * command line argument and re-discovers how it should behave.
+ */
+
+int cmd_main(int argc, const char **argv)
+{
+ int listen_port = 0;
+ struct string_list listen_addr = STRING_LIST_INIT_NODUP;
+ int worker_mode = 0;
+ int i;
+
+ trace2_cmd_name("test-http-server");
+ trace2_cmd_list_config();
+ trace2_cmd_list_env_vars();
+ setup_git_directory_gently(NULL);
+
+ for (i = 1; i < argc; i++) {
+ const char *arg = argv[i];
+ const char *v;
+
+ if (skip_prefix(arg, "--listen=", &v)) {
+ string_list_append(&listen_addr, xstrdup_tolower(v));
+ continue;
+ }
+ if (skip_prefix(arg, "--port=", &v)) {
+ char *end;
+ unsigned long n;
+ n = strtoul(v, &end, 0);
+ if (*v && !*end) {
+ listen_port = n;
+ continue;
+ }
+ }
+ if (!strcmp(arg, "--worker")) {
+ worker_mode = 1;
+ trace2_cmd_mode("worker");
+ continue;
+ }
+ if (!strcmp(arg, "--verbose")) {
+ verbose = 1;
+ continue;
+ }
+ if (skip_prefix(arg, "--timeout=", &v)) {
+ timeout = atoi(v);
+ continue;
+ }
+ if (skip_prefix(arg, "--max-connections=", &v)) {
+ max_connections = atoi(v);
+ if (max_connections < 0)
+ max_connections = 0; /* unlimited */
+ continue;
+ }
+ if (!strcmp(arg, "--reuseaddr")) {
+ reuseaddr = 1;
+ continue;
+ }
+ if (skip_prefix(arg, "--pid-file=", &v)) {
+ pid_file = v;
+ continue;
+ }
+
+ fprintf(stderr, "error: unknown argument '%s'\n", arg);
+ usage(test_http_auth_usage);
+ }
+
+ /* avoid splitting a message in the middle */
+ setvbuf(stderr, NULL, _IOFBF, 4096);
+
+ if (listen_port == 0)
+ listen_port = DEFAULT_GIT_PORT;
+
+ /*
+ * If no --listen=<addr> args are given, the setup_named_sock()
+ * code will use receive a NULL address and set INADDR_ANY.
+ * This exposes both internal and external interfaces on the
+ * port.
+ *
+ * Disallow that and default to the internal-use-only loopback
+ * address.
+ */
+ if (!listen_addr.nr)
+ string_list_append(&listen_addr, "127.0.0.1");
+
+ /*
+ * worker_mode is set in our own child process instances
+ * (that are bound to a connected socket from a client).
+ */
+ if (worker_mode)
+ return worker();
+
+ /*
+ * `cld_argv` is a bit of a clever hack. The top-level instance
+ * of test-http-server does the normal bind/listen/accept stuff.
+ * For each incoming socket, the top-level process spawns
+ * a child instance of test-http-server *WITH* the additional
+ * `--worker` argument. This causes the child to set `worker_mode`
+ * and immediately call `worker()` using the connected socket (and
+ * without the usual need for fork() or threads).
+ *
+ * The magic here is made possible because `cld_argv` is static
+ * and handle() (called by service_loop()) knows about it.
+ */
+ strvec_push(&cld_argv, argv[0]);
+ strvec_push(&cld_argv, "--worker");
+ for (i = 1; i < argc; ++i)
+ strvec_push(&cld_argv, argv[i]);
+
+ /*
+ * Setup primary instance to listen for connections.
+ */
+ return serve(&listen_addr, listen_port);
+}
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 01/12] daemon: libify socket setup and option functions
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Extract functions for setting up listening sockets and keep-alive options
from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct
dependencies on global state by inlining the behaviour at the callsites
for all libified functions.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
Makefile | 1 +
daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++
daemon-utils.h | 23 ++++++
daemon.c | 214 +------------------------------------------------
4 files changed, 237 insertions(+), 210 deletions(-)
create mode 100644 daemon-utils.c
create mode 100644 daemon-utils.h
diff --git a/Makefile b/Makefile
index b258fdbed86..2654094dbb5 100644
--- a/Makefile
+++ b/Makefile
@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o
LIB_OBJS += csum-file.o
LIB_OBJS += ctype.o
LIB_OBJS += date.o
+LIB_OBJS += daemon-utils.o
LIB_OBJS += decorate.o
LIB_OBJS += delta-islands.o
LIB_OBJS += diagnose.o
diff --git a/daemon-utils.c b/daemon-utils.c
new file mode 100644
index 00000000000..b96b55962db
--- /dev/null
+++ b/daemon-utils.c
@@ -0,0 +1,209 @@
+#include "cache.h"
+#include "daemon-utils.h"
+
+void set_keep_alive(int sockfd, log_fn logerror)
+{
+ int ka = 1;
+
+ if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
+ if (errno != ENOTSOCK)
+ logerror("unable to set SO_KEEPALIVE on socket: %s",
+ strerror(errno));
+ }
+}
+
+static int set_reuse_addr(int sockfd)
+{
+ int on = 1;
+
+ return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
+ &on, sizeof(on));
+}
+
+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
+{
+#ifdef NO_IPV6
+ static char ip[INET_ADDRSTRLEN];
+#else
+ static char ip[INET6_ADDRSTRLEN];
+#endif
+
+ switch (family) {
+#ifndef NO_IPV6
+ case AF_INET6:
+ inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
+ break;
+#endif
+ case AF_INET:
+ inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
+ break;
+ default:
+ xsnprintf(ip, sizeof(ip), "<unknown>");
+ }
+ return ip;
+}
+
+#ifndef NO_IPV6
+
+static int setup_named_sock(char *listen_addr, int listen_port,
+ struct socketlist *socklist, int reuseaddr,
+ log_fn logerror)
+{
+ int socknum = 0;
+ char pbuf[NI_MAXSERV];
+ struct addrinfo hints, *ai0, *ai;
+ int gai;
+ long flags;
+
+ xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
+ memset(&hints, 0, sizeof(hints));
+ hints.ai_family = AF_UNSPEC;
+ hints.ai_socktype = SOCK_STREAM;
+ hints.ai_protocol = IPPROTO_TCP;
+ hints.ai_flags = AI_PASSIVE;
+
+ gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
+ if (gai) {
+ logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
+ return 0;
+ }
+
+ for (ai = ai0; ai; ai = ai->ai_next) {
+ int sockfd;
+
+ sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
+ if (sockfd < 0)
+ continue;
+ if (sockfd >= FD_SETSIZE) {
+ logerror("Socket descriptor too large");
+ close(sockfd);
+ continue;
+ }
+
+#ifdef IPV6_V6ONLY
+ if (ai->ai_family == AF_INET6) {
+ int on = 1;
+ setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
+ &on, sizeof(on));
+ /* Note: error is not fatal */
+ }
+#endif
+
+ if (reuseaddr && set_reuse_addr(sockfd)) {
+ logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+ close(sockfd);
+ continue;
+ }
+
+ set_keep_alive(sockfd, logerror);
+
+ if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
+ logerror("Could not bind to %s: %s",
+ ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+ strerror(errno));
+ close(sockfd);
+ continue; /* not fatal */
+ }
+ if (listen(sockfd, 5) < 0) {
+ logerror("Could not listen to %s: %s",
+ ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
+ strerror(errno));
+ close(sockfd);
+ continue; /* not fatal */
+ }
+
+ flags = fcntl(sockfd, F_GETFD, 0);
+ if (flags >= 0)
+ fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+ ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+ socklist->list[socklist->nr++] = sockfd;
+ socknum++;
+ }
+
+ freeaddrinfo(ai0);
+
+ return socknum;
+}
+
+#else /* NO_IPV6 */
+
+static int setup_named_sock(char *listen_addr, int listen_port,
+ struct socketlist *socklist, int reuseaddr,
+ log_fn logerror)
+{
+ struct sockaddr_in sin;
+ int sockfd;
+ long flags;
+
+ memset(&sin, 0, sizeof sin);
+ sin.sin_family = AF_INET;
+ sin.sin_port = htons(listen_port);
+
+ if (listen_addr) {
+ /* Well, host better be an IP address here. */
+ if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
+ return 0;
+ } else {
+ sin.sin_addr.s_addr = htonl(INADDR_ANY);
+ }
+
+ sockfd = socket(AF_INET, SOCK_STREAM, 0);
+ if (sockfd < 0)
+ return 0;
+
+ if (reuseaddr && set_reuse_addr(sockfd)) {
+ logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
+ close(sockfd);
+ return 0;
+ }
+
+ set_keep_alive(sockfd, logerror);
+
+ if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
+ logerror("Could not bind to %s: %s",
+ ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+ strerror(errno));
+ close(sockfd);
+ return 0;
+ }
+
+ if (listen(sockfd, 5) < 0) {
+ logerror("Could not listen to %s: %s",
+ ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
+ strerror(errno));
+ close(sockfd);
+ return 0;
+ }
+
+ flags = fcntl(sockfd, F_GETFD, 0);
+ if (flags >= 0)
+ fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
+
+ ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
+ socklist->list[socklist->nr++] = sockfd;
+ return 1;
+}
+
+#endif
+
+void socksetup(struct string_list *listen_addr, int listen_port,
+ struct socketlist *socklist, int reuseaddr,
+ log_fn logerror)
+{
+ if (!listen_addr->nr)
+ setup_named_sock(NULL, listen_port, socklist, reuseaddr,
+ logerror);
+ else {
+ int i, socknum;
+ for (i = 0; i < listen_addr->nr; i++) {
+ socknum = setup_named_sock(listen_addr->items[i].string,
+ listen_port, socklist, reuseaddr,
+ logerror);
+
+ if (socknum == 0)
+ logerror("unable to allocate any listen sockets for host %s on port %u",
+ listen_addr->items[i].string, listen_port);
+ }
+ }
+}
diff --git a/daemon-utils.h b/daemon-utils.h
new file mode 100644
index 00000000000..6710a2a6dc0
--- /dev/null
+++ b/daemon-utils.h
@@ -0,0 +1,23 @@
+#ifndef DAEMON_UTILS_H
+#define DAEMON_UTILS_H
+
+#include "git-compat-util.h"
+#include "string-list.h"
+
+typedef void (*log_fn)(const char *msg, ...);
+
+struct socketlist {
+ int *list;
+ size_t nr;
+ size_t alloc;
+};
+
+/* Enable sending of keep-alive messages on the socket. */
+void set_keep_alive(int sockfd, log_fn logerror);
+
+/* Setup a number of sockets to listen on the provided addresses. */
+void socksetup(struct string_list *listen_addr, int listen_port,
+ struct socketlist *socklist, int reuseaddr,
+ log_fn logerror);
+
+#endif
diff --git a/daemon.c b/daemon.c
index 0ae7d12b5c1..1ed4e705680 100644
--- a/daemon.c
+++ b/daemon.c
@@ -1,9 +1,9 @@
#include "cache.h"
#include "config.h"
+#include "daemon-utils.h"
#include "pkt-line.h"
#include "run-command.h"
#include "strbuf.h"
-#include "string-list.h"
#ifdef NO_INITGROUPS
#define initgroups(x, y) (0) /* nothing */
@@ -737,17 +737,6 @@ static void hostinfo_clear(struct hostinfo *hi)
strbuf_release(&hi->tcp_port);
}
-static void set_keep_alive(int sockfd)
-{
- int ka = 1;
-
- if (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {
- if (errno != ENOTSOCK)
- logerror("unable to set SO_KEEPALIVE on socket: %s",
- strerror(errno));
- }
-}
-
static int execute(void)
{
char *line = packet_buffer;
@@ -759,7 +748,7 @@ static int execute(void)
if (addr)
loginfo("Connection from %s:%s", addr, port);
- set_keep_alive(0);
+ set_keep_alive(0, logerror);
alarm(init_timeout ? init_timeout : timeout);
pktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);
alarm(0);
@@ -938,202 +927,6 @@ static void child_handler(int signo)
signal(SIGCHLD, child_handler);
}
-static int set_reuse_addr(int sockfd)
-{
- int on = 1;
-
- if (!reuseaddr)
- return 0;
- return setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,
- &on, sizeof(on));
-}
-
-struct socketlist {
- int *list;
- size_t nr;
- size_t alloc;
-};
-
-static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)
-{
-#ifdef NO_IPV6
- static char ip[INET_ADDRSTRLEN];
-#else
- static char ip[INET6_ADDRSTRLEN];
-#endif
-
- switch (family) {
-#ifndef NO_IPV6
- case AF_INET6:
- inet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);
- break;
-#endif
- case AF_INET:
- inet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);
- break;
- default:
- xsnprintf(ip, sizeof(ip), "<unknown>");
- }
- return ip;
-}
-
-#ifndef NO_IPV6
-
-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-{
- int socknum = 0;
- char pbuf[NI_MAXSERV];
- struct addrinfo hints, *ai0, *ai;
- int gai;
- long flags;
-
- xsnprintf(pbuf, sizeof(pbuf), "%d", listen_port);
- memset(&hints, 0, sizeof(hints));
- hints.ai_family = AF_UNSPEC;
- hints.ai_socktype = SOCK_STREAM;
- hints.ai_protocol = IPPROTO_TCP;
- hints.ai_flags = AI_PASSIVE;
-
- gai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);
- if (gai) {
- logerror("getaddrinfo() for %s failed: %s", listen_addr, gai_strerror(gai));
- return 0;
- }
-
- for (ai = ai0; ai; ai = ai->ai_next) {
- int sockfd;
-
- sockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);
- if (sockfd < 0)
- continue;
- if (sockfd >= FD_SETSIZE) {
- logerror("Socket descriptor too large");
- close(sockfd);
- continue;
- }
-
-#ifdef IPV6_V6ONLY
- if (ai->ai_family == AF_INET6) {
- int on = 1;
- setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,
- &on, sizeof(on));
- /* Note: error is not fatal */
- }
-#endif
-
- if (set_reuse_addr(sockfd)) {
- logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
- close(sockfd);
- continue;
- }
-
- set_keep_alive(sockfd);
-
- if (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {
- logerror("Could not bind to %s: %s",
- ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
- strerror(errno));
- close(sockfd);
- continue; /* not fatal */
- }
- if (listen(sockfd, 5) < 0) {
- logerror("Could not listen to %s: %s",
- ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),
- strerror(errno));
- close(sockfd);
- continue; /* not fatal */
- }
-
- flags = fcntl(sockfd, F_GETFD, 0);
- if (flags >= 0)
- fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-
- ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
- socklist->list[socklist->nr++] = sockfd;
- socknum++;
- }
-
- freeaddrinfo(ai0);
-
- return socknum;
-}
-
-#else /* NO_IPV6 */
-
-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)
-{
- struct sockaddr_in sin;
- int sockfd;
- long flags;
-
- memset(&sin, 0, sizeof sin);
- sin.sin_family = AF_INET;
- sin.sin_port = htons(listen_port);
-
- if (listen_addr) {
- /* Well, host better be an IP address here. */
- if (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)
- return 0;
- } else {
- sin.sin_addr.s_addr = htonl(INADDR_ANY);
- }
-
- sockfd = socket(AF_INET, SOCK_STREAM, 0);
- if (sockfd < 0)
- return 0;
-
- if (set_reuse_addr(sockfd)) {
- logerror("Could not set SO_REUSEADDR: %s", strerror(errno));
- close(sockfd);
- return 0;
- }
-
- set_keep_alive(sockfd);
-
- if ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {
- logerror("Could not bind to %s: %s",
- ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
- strerror(errno));
- close(sockfd);
- return 0;
- }
-
- if (listen(sockfd, 5) < 0) {
- logerror("Could not listen to %s: %s",
- ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),
- strerror(errno));
- close(sockfd);
- return 0;
- }
-
- flags = fcntl(sockfd, F_GETFD, 0);
- if (flags >= 0)
- fcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);
-
- ALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);
- socklist->list[socklist->nr++] = sockfd;
- return 1;
-}
-
-#endif
-
-static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)
-{
- if (!listen_addr->nr)
- setup_named_sock(NULL, listen_port, socklist);
- else {
- int i, socknum;
- for (i = 0; i < listen_addr->nr; i++) {
- socknum = setup_named_sock(listen_addr->items[i].string,
- listen_port, socklist);
-
- if (socknum == 0)
- logerror("unable to allocate any listen sockets for host %s on port %u",
- listen_addr->items[i].string, listen_port);
- }
- }
-}
-
static int service_loop(struct socketlist *socklist)
{
struct pollfd *pfd;
@@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port,
{
struct socketlist socklist = { NULL, 0, 0 };
- socksetup(listen_addr, listen_port, &socklist);
+ socksetup(listen_addr, listen_port, &socklist, reuseaddr,
+ logerror);
if (socklist.nr == 0)
die("unable to allocate any listen sockets on port %u",
listen_port);
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 06/12] test-http-server: add HTTP request parsing
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Add ability to parse HTTP requests to the test-http-server test helper.
Introduce `struct req` to store request information including:
* HTTP method & version
* Request path and query parameters
* Headers
* Content type and length (from `Content-Type` and `-Length` headers)
Failure to parse the request results in a 400 Bad Request response to
the client. Note that we're not trying to support all possible requests
here, but just enough to exercise all code under test.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
t/helper/test-http-server.c | 202 +++++++++++++++++++++++++++++++++++-
t/t5556-http-auth.sh | 90 ++++++++++++++++
2 files changed, 290 insertions(+), 2 deletions(-)
create mode 100755 t/t5556-http-auth.sh
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 7ca4ddc7999..900f5733cc1 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -81,8 +81,53 @@ enum worker_result {
* Close the socket and clean up. Does not imply an error.
*/
WR_HANGUP = 2,
+
+ /*
+ * Unexpected request message or error in request parsing.
+ * Respond with an 400 error. Close the socket and cleanup.
+ * Exit child-process with a non-zero status.
+ */
+ WR_CLIENT_ERROR = 3,
+};
+
+/*
+ * Fields from a parsed HTTP request.
+ */
+struct req {
+ struct strbuf start_line;
+
+ const char *method;
+ const char *http_version;
+
+ struct strbuf uri_path;
+ struct strbuf query_args;
+
+ struct string_list header_list;
+ const char *content_type;
+ uintmax_t content_length;
+ unsigned has_content_length:1;
};
+#define REQ__INIT { \
+ .start_line = STRBUF_INIT, \
+ .uri_path = STRBUF_INIT, \
+ .query_args = STRBUF_INIT, \
+ .header_list = STRING_LIST_INIT_NODUP, \
+ .content_type = NULL, \
+ .content_length = 0, \
+ .has_content_length = 0, \
+}
+
+static void req__release(struct req *req)
+{
+ strbuf_release(&req->start_line);
+
+ strbuf_release(&req->uri_path);
+ strbuf_release(&req->query_args);
+
+ string_list_clear(&req->header_list, 0);
+}
+
static enum worker_result send_http_error(int fd, int http_code,
const char *http_code_name,
int retry_after_seconds,
@@ -143,8 +188,150 @@ done:
return wr;
}
+/*
+ * Read the HTTP request up to the start of the optional message-body.
+ * We do this byte-by-byte because we have keep-alive turned on and
+ * cannot rely on an EOF.
+ *
+ * https://tools.ietf.org/html/rfc7230
+ *
+ * We cannot call die() here because our caller needs to properly
+ * respond to the client and/or close the socket before this
+ * child exits so that the client doesn't get a connection reset
+ * by peer error.
+ */
+static enum worker_result req__read(struct req *req, int fd)
+{
+ struct strbuf h = STRBUF_INIT;
+ struct string_list start_line_fields = STRING_LIST_INIT_DUP;
+ int nr_start_line_fields;
+ const char *uri_target;
+ const char *query;
+ char *hp;
+ const char *hv;
+
+ enum worker_result result = WR_OK;
+
+ /*
+ * Read line 0 of the request and split it into component parts:
+ *
+ * <method> SP <uri-target> SP <HTTP-version> CRLF
+ *
+ */
+ if (strbuf_getwholeline_fd(&req->start_line, fd, '\n') == EOF) {
+ result = WR_HANGUP;
+ goto done;
+ }
+
+ strbuf_trim_trailing_newline(&req->start_line);
+
+ nr_start_line_fields = string_list_split(&start_line_fields,
+ req->start_line.buf,
+ ' ', -1);
+ if (nr_start_line_fields != 3) {
+ logerror("could not parse request start-line '%s'",
+ req->start_line.buf);
+ result = WR_CLIENT_ERROR;
+ goto done;
+ }
+
+ req->method = xstrdup(start_line_fields.items[0].string);
+ req->http_version = xstrdup(start_line_fields.items[2].string);
+
+ uri_target = start_line_fields.items[1].string;
+
+ if (strcmp(req->http_version, "HTTP/1.1")) {
+ logerror("unsupported version '%s' (expecting HTTP/1.1)",
+ req->http_version);
+ result = WR_CLIENT_ERROR;
+ goto done;
+ }
+
+ query = strchr(uri_target, '?');
+
+ if (query) {
+ strbuf_add(&req->uri_path, uri_target, (query - uri_target));
+ strbuf_trim_trailing_dir_sep(&req->uri_path);
+ strbuf_addstr(&req->query_args, query + 1);
+ } else {
+ strbuf_addstr(&req->uri_path, uri_target);
+ strbuf_trim_trailing_dir_sep(&req->uri_path);
+ }
+
+ /*
+ * Read the set of HTTP headers into a string-list.
+ */
+ while (1) {
+ if (strbuf_getwholeline_fd(&h, fd, '\n') == EOF)
+ goto done;
+ strbuf_trim_trailing_newline(&h);
+
+ if (!h.len)
+ goto done; /* a blank line ends the header */
+
+ hp = strbuf_detach(&h, NULL);
+ string_list_append(&req->header_list, hp);
+
+ /* also store common request headers as struct req members */
+ if (skip_iprefix(hp, "Content-Type: ", &hv)) {
+ req->content_type = hv;
+ } else if (skip_iprefix(hp, "Content-Length: ", &hv)) {
+ /*
+ * Content-Length is always non-negative, but has no
+ * upper bound according to RFC 7230 (§3.3.2).
+ */
+ intmax_t len = 0;
+ if (sscanf(hv, "%"PRIdMAX, &len) != 1 || len < 0 ||
+ len == INTMAX_MAX) {
+ logerror("invalid content-length: '%s'", hv);
+ result = WR_CLIENT_ERROR;
+ goto done;
+ }
+
+ req->content_length = (uintmax_t)len;
+ req->has_content_length = 1;
+ }
+ }
+
+ /*
+ * We do not attempt to read the <message-body>, if it exists.
+ * We let our caller read/chunk it in as appropriate.
+ */
+
+done:
+ string_list_clear(&start_line_fields, 0);
+
+ /*
+ * This is useful for debugging the request, but very noisy.
+ */
+ if (trace2_is_enabled()) {
+ struct string_list_item *item;
+ trace2_printf("%s: %s", TR2_CAT, req->start_line.buf);
+ trace2_printf("%s: hver: %s", TR2_CAT, req->http_version);
+ trace2_printf("%s: hmth: %s", TR2_CAT, req->method);
+ trace2_printf("%s: path: %s", TR2_CAT, req->uri_path.buf);
+ trace2_printf("%s: qury: %s", TR2_CAT, req->query_args.buf);
+ if (req->has_content_length)
+ trace2_printf("%s: clen: %"PRIuMAX, TR2_CAT,
+ req->content_length);
+ if (req->content_type)
+ trace2_printf("%s: ctyp: %s", TR2_CAT, req->content_type);
+ for_each_string_list_item(item, &req->header_list)
+ trace2_printf("%s: hdrs: %s", TR2_CAT, item->string);
+ }
+
+ return result;
+}
+
+static enum worker_result dispatch(struct req *req)
+{
+ return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
+ WR_HANGUP);
+}
+
static enum worker_result worker(void)
{
+ struct req req = REQ__INIT;
char *client_addr = getenv("REMOTE_ADDR");
char *client_port = getenv("REMOTE_PORT");
enum worker_result wr = WR_OK;
@@ -155,9 +342,20 @@ static enum worker_result worker(void)
set_keep_alive(0, logerror);
while (1) {
- wr = send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1,
- NULL, WR_HANGUP);
+ req__release(&req);
+
+ alarm(timeout);
+ wr = req__read(&req, 0);
+ alarm(0);
+
+ if (wr == WR_CLIENT_ERROR)
+ wr = send_http_error(STDOUT_FILENO, 400, "Bad Request",
+ -1, NULL, wr);
+
+ if (wr != WR_OK)
+ break;
+ wr = dispatch(&req);
if (wr != WR_OK)
break;
}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
new file mode 100755
index 00000000000..06efc85ca53
--- /dev/null
+++ b/t/t5556-http-auth.sh
@@ -0,0 +1,90 @@
+#!/bin/sh
+
+test_description='test http auth header and credential helper interop'
+
+TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
+
+# Setup a repository
+#
+REPO_DIR="$TRASH_DIRECTORY"/repo
+
+SERVER_LOG="$TRASH_DIRECTORY"/OUT.server.log
+
+PATH="$GIT_BUILD_DIR/t/helper/:$PATH" && export PATH
+
+test_expect_success 'setup repos' '
+ test_create_repo "$REPO_DIR" &&
+ git -C "$REPO_DIR" branch -M main
+'
+
+run_http_server_worker() {
+ (
+ cd "$REPO_DIR"
+ test-http-server --worker "$@" 2>"$SERVER_LOG" | tr -d "\r"
+ )
+}
+
+per_test_cleanup () {
+ rm -f OUT.* &&
+ rm -f IN.* &&
+}
+
+test_expect_success 'http auth server request parsing' '
+ test_when_finished "per_test_cleanup" &&
+
+ cat >auth.config <<-EOF &&
+ [auth]
+ allowAnonymous = true
+ EOF
+
+ echo "HTTP/1.1 400 Bad Request" >OUT.http400 &&
+ echo "HTTP/1.1 200 OK" >OUT.http200 &&
+
+ cat >IN.http.valid <<-EOF &&
+ GET /info/refs HTTP/1.1
+ Content-Length: 0
+ EOF
+
+ cat >IN.http.badfirstline <<-EOF &&
+ /info/refs GET HTTP
+ EOF
+
+ cat >IN.http.badhttpver <<-EOF &&
+ GET /info/refs HTTP/999.9
+ EOF
+
+ cat >IN.http.ltzlen <<-EOF &&
+ GET /info/refs HTTP/1.1
+ Content-Length: -1
+ EOF
+
+ cat >IN.http.badlen <<-EOF &&
+ GET /info/refs HTTP/1.1
+ Content-Length: not-a-number
+ EOF
+
+ cat >IN.http.overlen <<-EOF &&
+ GET /info/refs HTTP/1.1
+ Content-Length: 9223372036854775807
+ EOF
+
+ run_http_server_worker \
+ --auth-config="$TRASH_DIRECTORY/auth.config" <IN.http.valid \
+ | head -n1 >OUT.actual &&
+ test_cmp OUT.http200 OUT.actual &&
+
+ run_http_server_worker <IN.http.badfirstline | head -n1 >OUT.actual &&
+ test_cmp OUT.http400 OUT.actual &&
+
+ run_http_server_worker <IN.http.ltzlen | head -n1 >OUT.actual &&
+ test_cmp OUT.http400 OUT.actual &&
+
+ run_http_server_worker <IN.http.badlen | head -n1 >OUT.actual &&
+ test_cmp OUT.http400 OUT.actual &&
+
+ run_http_server_worker <IN.http.overlen | head -n1 >OUT.actual &&
+ test_cmp OUT.http400 OUT.actual
+'
+
+test_done
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 08/12] test-http-server: add simple authentication
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Add simple authentication to the test-http-server test helper.
Authentication schemes and sets of valid tokens can be specified via
a configuration file (in the normal gitconfig file format).
Incoming requests are compared against the set of valid schemes and
tokens and only approved if a matching token is found, or if no auth
was provided and anonymous auth is enabled.
Configuration for auth includes a simple set of three options:
[auth]
challenge = <scheme>[:<challenge_params>]
token = <scheme>:[<token>]*
allowAnonymous = <bool>
`auth.challenge` allows you define what authentication schemes, and
optional challenge parameters the server should use. Scheme names are
unique and subsequently specified challenge parameters in the config
file will replace previously specified ones.
`auth.token` allows you to define the set of value token values for an
authentication scheme. This is a multi-var and each entry in the
config file will append to the set of valid tokens for that scheme.
Specifying an empty token value will clear the list of tokens so far for
that scheme, i.e. `token = <scheme>:`.
`auth.allowAnonymous` controls whether or not unauthenticated requests
(those without any `Authorization` headers) should succeed or not, and
trigger a 401 Unauthorized response.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
t/helper/test-http-server.c | 232 +++++++++++++++++++++++++++++++++++-
t/t5556-http-auth.sh | 43 ++++++-
2 files changed, 272 insertions(+), 3 deletions(-)
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 4191daf3c64..72c6cca7e5c 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -7,6 +7,7 @@
#include "version.h"
#include "dir.h"
#include "date.h"
+#include "config.h"
#define TR2_CAT "test-http-server"
@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =
" [--timeout=<n>] [--max-connections=<n>]\n"
" [--reuseaddr] [--pid-file=<file>]\n"
" [--listen=<host_or_ipaddr>]* [--port=<n>]\n"
+" [--auth-config=<file>]\n"
;
static unsigned int timeout;
@@ -349,7 +351,7 @@ static int is_git_request(struct req *req)
!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);
}
-static enum worker_result do__git(struct req *req)
+static enum worker_result do__git(struct req *req, const char *user)
{
const char *ok = "HTTP/1.1 200 OK\r\n";
struct child_process cp = CHILD_PROCESS_INIT;
@@ -366,10 +368,16 @@ static enum worker_result do__git(struct req *req)
* exit status of the process, then write the HTTP status line followed
* by the http-backend output. This is outside of the scope of this test
* helper's use at time of writing.
+ *
+ * The important auth responses (401) we are handling prior to getting
+ * to this point.
*/
if (write(STDOUT_FILENO, ok, strlen(ok)) < 0)
return error(_("could not send '%s'"), ok);
+ if (user)
+ strvec_pushf(&cp.env, "REMOTE_USER=%s", user);
+
strvec_pushf(&cp.env, "REQUEST_METHOD=%s", req->method);
strvec_pushf(&cp.env, "PATH_TRANSLATED=%s", req->uri_path.buf);
strvec_push(&cp.env, "SERVER_PROTOCOL=HTTP/1.1");
@@ -388,10 +396,217 @@ static enum worker_result do__git(struct req *req)
return !!res;
}
+enum auth_result {
+ /* No auth module matches the request. */
+ AUTH_UNKNOWN = 0,
+
+ /* Auth module denied the request. */
+ AUTH_DENY = 1,
+
+ /* Auth module successfully validated the request. */
+ AUTH_ALLOW = 2,
+};
+
+struct auth_module {
+ char *scheme;
+ char *challenge_params;
+ struct string_list *tokens;
+};
+
+static int allow_anonymous;
+static struct auth_module **auth_modules = NULL;
+static size_t auth_modules_nr = 0;
+static size_t auth_modules_alloc = 0;
+
+static struct auth_module *get_auth_module(const char *scheme, int create)
+{
+ struct auth_module *mod;
+ for (size_t i = 0; i < auth_modules_nr; i++) {
+ mod = auth_modules[i];
+ if (!strcasecmp(mod->scheme, scheme))
+ return mod;
+ }
+
+ if (create) {
+ struct auth_module *mod = xmalloc(sizeof(struct auth_module));
+ mod->scheme = xstrdup(scheme);
+ mod->challenge_params = NULL;
+ ALLOC_ARRAY(mod->tokens, 1);
+ string_list_init_dup(mod->tokens);
+
+ ALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);
+ auth_modules[auth_modules_nr++] = mod;
+
+ return mod;
+ }
+
+ return NULL;
+}
+
+static int is_authed(struct req *req, const char **user, enum worker_result *wr)
+{
+ enum auth_result result = AUTH_UNKNOWN;
+ struct string_list hdrs = STRING_LIST_INIT_NODUP;
+ struct auth_module *mod;
+
+ struct string_list_item *hdr;
+ struct string_list_item *token;
+ const char *v;
+ struct strbuf **split = NULL;
+ int i;
+ char *challenge;
+
+ /*
+ * Check all auth modules and try to validate the request.
+ * The first Authorization header that matches a known auth module
+ * scheme will be consulted to either approve or deny the request.
+ * If no module is found, or if there is no valid token, then 401 error.
+ * Otherwise, only permit the request if anonymous auth is enabled.
+ * It's atypical for user agents/clients to send multiple Authorization
+ * headers, but not explicitly forbidden or defined.
+ */
+ for_each_string_list_item(hdr, &req->header_list) {
+ if (skip_iprefix(hdr->string, "Authorization: ", &v)) {
+ split = strbuf_split_str(v, ' ', 2);
+ if (split[0] && split[1]) {
+ /* trim trailing space ' ' */
+ strbuf_rtrim(split[0]);
+
+ mod = get_auth_module(split[0]->buf, 0);
+ if (mod) {
+ result = AUTH_DENY;
+
+ for_each_string_list_item(token, mod->tokens) {
+ if (!strcmp(split[1]->buf, token->string)) {
+ result = AUTH_ALLOW;
+ break;
+ }
+ }
+
+ strbuf_list_free(split);
+ goto done;
+ }
+ }
+
+ strbuf_list_free(split);
+ }
+ }
+
+done:
+ switch (result) {
+ case AUTH_ALLOW:
+ trace2_printf("%s: auth '%s' ALLOW", TR2_CAT, mod->scheme);
+ *user = "VALID_TEST_USER";
+ *wr = WR_OK;
+ break;
+
+ case AUTH_DENY:
+ trace2_printf("%s: auth '%s' DENY", TR2_CAT, mod->scheme);
+ /* fall-through */
+
+ case AUTH_UNKNOWN:
+ if (result != AUTH_DENY && allow_anonymous)
+ break;
+
+ for (i = 0; i < auth_modules_nr; i++) {
+ mod = auth_modules[i];
+ if (mod->challenge_params)
+ challenge = xstrfmt("WWW-Authenticate: %s %s",
+ mod->scheme,
+ mod->challenge_params);
+ else
+ challenge = xstrfmt("WWW-Authenticate: %s",
+ mod->scheme);
+ string_list_append(&hdrs, challenge);
+ }
+
+ *wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
+ &hdrs, *wr);
+ }
+
+ string_list_clear(&hdrs, 0);
+
+ return result == AUTH_ALLOW ||
+ (result == AUTH_UNKNOWN && allow_anonymous);
+}
+
+static int split_auth_param(const char *str, char **scheme, char **val)
+{
+ struct strbuf **p = strbuf_split_str(str, ':', 2);
+
+ if (!p[0])
+ return -1;
+
+ /* trim trailing ':' */
+ if (p[0]->len && p[0]->buf[p[0]->len - 1] == ':')
+ strbuf_setlen(p[0], p[0]->len - 1);
+
+ *scheme = strbuf_detach(p[0], NULL);
+ *val = p[1] ? strbuf_detach(p[1], NULL) : NULL;
+
+ strbuf_list_free(p);
+ return 0;
+}
+
+static int read_auth_config(const char *name, const char *val, void *data)
+{
+ int ret = 0;
+ char *scheme = NULL;
+ char *token = NULL;
+ char *challenge = NULL;
+ struct auth_module *mod;
+
+ if (!strcmp(name, "auth.challenge")) {
+ if (split_auth_param(val, &scheme, &challenge)) {
+ ret = error("invalid auth challenge '%s'", val);
+ goto cleanup;
+ }
+
+ mod = get_auth_module(scheme, 1);
+
+ /* Replace any existing challenge parameters */
+ free(mod->challenge_params);
+ mod->challenge_params = challenge ? xstrdup(challenge) : NULL;
+ } else if (!strcmp(name, "auth.token")) {
+ if (split_auth_param(val, &scheme, &token)) {
+ ret = error("invalid auth token '%s'", val);
+ goto cleanup;
+ }
+
+ mod = get_auth_module(scheme, 1);
+
+ /*
+ * Append to set of valid tokens unless an empty token value
+ * is provided, then clear the existing list.
+ */
+ if (token)
+ string_list_append(mod->tokens, token);
+ else
+ string_list_clear(mod->tokens, 1);
+ } else if (!strcmp(name, "auth.allowanonymous")) {
+ allow_anonymous = git_config_bool(name, val);
+ } else {
+ warning("unknown auth config '%s'", name);
+ }
+
+cleanup:
+ free(scheme);
+ free(token);
+ free(challenge);
+
+ return ret;
+}
+
static enum worker_result dispatch(struct req *req)
{
+ enum worker_result wr = WR_OK;
+ const char *user = NULL;
+
+ if (!is_authed(req, &user, &wr))
+ return wr;
+
if (is_git_request(req))
- return do__git(req);
+ return do__git(req, user);
return send_http_error(STDOUT_FILENO, 501, "Not Implemented", -1, NULL,
WR_HANGUP);
@@ -655,6 +870,19 @@ int cmd_main(int argc, const char **argv)
pid_file = v;
continue;
}
+ if (skip_prefix(arg, "--auth-config=", &v)) {
+ if (!strlen(v)) {
+ error("invalid argument - missing file path");
+ usage(test_http_auth_usage);
+ }
+
+ if (git_config_from_file(read_auth_config, v, NULL)) {
+ error("failed to read auth config file '%s'", v);
+ usage(test_http_auth_usage);
+ }
+
+ continue;
+ }
fprintf(stderr, "error: unknown argument '%s'\n", arg);
usage(test_http_auth_usage);
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index c0a47ce342b..20fd9b09aef 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -101,6 +101,7 @@ per_test_cleanup () {
stop_http_server &&
rm -f OUT.* &&
rm -f IN.* &&
+ rm -f auth.config
}
test_expect_success 'http auth server request parsing' '
@@ -160,11 +161,51 @@ test_expect_success 'http auth server request parsing' '
test_cmp OUT.http400 OUT.actual
'
+test_expect_success CURL 'http auth server auth config' '
+ test_when_finished "per_test_cleanup" &&
+
+ cat >auth.config <<-EOF &&
+ [auth]
+ challenge = no-params
+ challenge = with-params:foo=\"bar\" p=1
+ challenge = with-params:foo=\"replaced\" q=1
+
+ token = no-explicit-challenge:valid-token
+ token = no-explicit-challenge:also-valid
+ token = reset-tokens:these-tokens
+ token = reset-tokens:will-be-reset
+ token = reset-tokens:
+ token = reset-tokens:the-only-valid-one
+
+ allowAnonymous = false
+ EOF
+
+ cat >OUT.expected <<-EOF &&
+ WWW-Authenticate: no-params
+ WWW-Authenticate: with-params foo="replaced" q=1
+ WWW-Authenticate: no-explicit-challenge
+ WWW-Authenticate: reset-tokens
+
+ Error: 401 Unauthorized
+ EOF
+
+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
+
+ curl --include $ORIGIN_URL >OUT.curl &&
+ tr -d "\r" <OUT.curl | sed -n "/WWW-Authenticate/,\$p" >OUT.actual &&
+
+ test_cmp OUT.expected OUT.actual
+'
test_expect_success 'http auth anonymous no challenge' '
test_when_finished "per_test_cleanup" &&
- start_http_server &&
+ cat >auth.config <<-EOF &&
+ [auth]
+ allowAnonymous = true
+ EOF
+
+ start_http_server --auth-config="$TRASH_DIRECTORY/auth.config" &&
# Attempt to read from a protected repository
git ls-remote $ORIGIN_URL
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 09/12] test-http-server: add sending of arbitrary headers
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Add the ability to send arbitrary headers in HTTP responses from the
test-http-server. This is useful when we want to test 'malformed'
response message handling.
Add the following option to the server auth config file:
[auth]
extraHeader = [<value>]*
Each `auth.extraHeader` value will be appended to the response headers
verbatim.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
t/helper/test-http-server.c | 6 ++++++
t/t5556-http-auth.sh | 7 +++++++
2 files changed, 13 insertions(+)
diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c
index 72c6cca7e5c..70bf15c3fa1 100644
--- a/t/helper/test-http-server.c
+++ b/t/helper/test-http-server.c
@@ -417,6 +417,7 @@ static int allow_anonymous;
static struct auth_module **auth_modules = NULL;
static size_t auth_modules_nr = 0;
static size_t auth_modules_alloc = 0;
+static struct strvec extra_headers = STRVEC_INIT;
static struct auth_module *get_auth_module(const char *scheme, int create)
{
@@ -520,6 +521,9 @@ done:
string_list_append(&hdrs, challenge);
}
+ for (i = 0; i < extra_headers.nr; i++)
+ string_list_append(&hdrs, extra_headers.v[i]);
+
*wr = send_http_error(STDOUT_FILENO, 401, "Unauthorized", -1,
&hdrs, *wr);
}
@@ -585,6 +589,8 @@ static int read_auth_config(const char *name, const char *val, void *data)
string_list_clear(mod->tokens, 1);
} else if (!strcmp(name, "auth.allowanonymous")) {
allow_anonymous = git_config_bool(name, val);
+ } else if (!strcmp(name, "auth.extraheader")) {
+ strvec_push(&extra_headers, val);
} else {
warning("unknown auth config '%s'", name);
}
diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh
index 20fd9b09aef..2c16c8f72a5 100755
--- a/t/t5556-http-auth.sh
+++ b/t/t5556-http-auth.sh
@@ -178,6 +178,10 @@ test_expect_success CURL 'http auth server auth config' '
token = reset-tokens:the-only-valid-one
allowAnonymous = false
+
+ extraHeader = X-Extra-Header: abc
+ extraHeader = X-Extra-Header: 123
+ extraHeader = X-Another: header\twith\twhitespace!
EOF
cat >OUT.expected <<-EOF &&
@@ -185,6 +189,9 @@ test_expect_success CURL 'http auth server auth config' '
WWW-Authenticate: with-params foo="replaced" q=1
WWW-Authenticate: no-explicit-challenge
WWW-Authenticate: reset-tokens
+ X-Extra-Header: abc
+ X-Extra-Header: 123
+ X-Another: header with whitespace!
Error: 401 Unauthorized
EOF
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Read and store the HTTP WWW-Authenticate response headers made for
a particular request.
This will allow us to pass important authentication challenge
information to credential helpers or others that would otherwise have
been lost.
According to RFC2616 Section 4.2 [1], header field names are not
case-sensitive meaning when collecting multiple values for the same
field name, we can just use the case of the first observed instance of
each field name and no normalisation is required.
libcurl only provides us with the ability to read all headers recieved
for a particular request, including any intermediate redirect requests
or proxies. The lines returned by libcurl include HTTP status lines
delinating any intermediate requests such as "HTTP/1.1 200". We use
these lines to reset the strvec of WWW-Authenticate header values as
we encounter them in order to only capture the final response headers.
The collection of all header values matching the WWW-Authenticate
header is complicated by the fact that it is legal for header fields to
be continued over multiple lines, but libcurl only gives us one line at
a time.
In the future [2] we may be able to leverage functions to read headers
from libcurl itself, but as of today we must do this ourselves.
[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2
[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
credential.c | 1 +
credential.h | 15 +++++++++
http.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 110 insertions(+)
diff --git a/credential.c b/credential.c
index f6389a50684..897b4679333 100644
--- a/credential.c
+++ b/credential.c
@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)
free(c->username);
free(c->password);
string_list_clear(&c->helpers, 0);
+ strvec_clear(&c->wwwauth_headers);
credential_init(c);
}
diff --git a/credential.h b/credential.h
index f430e77fea4..6f2e5bc610b 100644
--- a/credential.h
+++ b/credential.h
@@ -2,6 +2,7 @@
#define CREDENTIAL_H
#include "string-list.h"
+#include "strvec.h"
/**
* The credentials API provides an abstracted way of gathering username and
@@ -115,6 +116,19 @@ struct credential {
*/
struct string_list helpers;
+ /**
+ * A `strvec` of WWW-Authenticate header values. Each string
+ * is the value of a WWW-Authenticate header in an HTTP response,
+ * in the order they were received in the response.
+ */
+ struct strvec wwwauth_headers;
+
+ /**
+ * Internal use only. Used to keep track of split header fields
+ * in order to fold multiple lines into one value.
+ */
+ unsigned header_is_last_match:1;
+
unsigned approved:1,
configured:1,
quit:1,
@@ -130,6 +144,7 @@ struct credential {
#define CREDENTIAL_INIT { \
.helpers = STRING_LIST_INIT_DUP, \
+ .wwwauth_headers = STRVEC_INIT, \
}
/* Initialize a credential structure, setting all fields to empty. */
diff --git a/http.c b/http.c
index a2a80318bb2..595c93bc7a3 100644
--- a/http.c
+++ b/http.c
@@ -183,6 +183,98 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
return nmemb;
}
+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)
+{
+ size_t size = st_mult(eltsize, nmemb);
+ struct strvec *values = &http_auth.wwwauth_headers;
+ struct strbuf buf = STRBUF_INIT;
+ const char *val;
+
+ /*
+ * Header lines may not come NULL-terminated from libcurl so we must
+ * limit all scans to the maximum length of the header line, or leverage
+ * strbufs for all operations.
+ *
+ * In addition, it is possible that header values can be split over
+ * multiple lines as per RFC 2616 (even though this has since been
+ * deprecated in RFC 7230). A continuation header field value is
+ * identified as starting with a space or horizontal tab.
+ *
+ * The formal definition of a header field as given in RFC 2616 is:
+ *
+ * message-header = field-name ":" [ field-value ]
+ * field-name = token
+ * field-value = *( field-content | LWS )
+ * field-content = <the OCTETs making up the field-value
+ * and consisting of either *TEXT or combinations
+ * of token, separators, and quoted-string>
+ */
+
+ strbuf_add(&buf, ptr, size);
+
+ /* Strip the CRLF that should be present at the end of each field */
+ strbuf_trim_trailing_newline(&buf);
+
+ /* Start of a new WWW-Authenticate header */
+ if (skip_iprefix(buf.buf, "www-authenticate:", &val)) {
+ while (isspace(*val))
+ val++;
+
+ strvec_push(values, val);
+ http_auth.header_is_last_match = 1;
+ goto exit;
+ }
+
+ /*
+ * This line could be a continuation of the previously matched header
+ * field. If this is the case then we should append this value to the
+ * end of the previously consumed value.
+ * Continuation lines start with at least one whitespace, maybe more,
+ * so we should collapse these down to a single SP (valid per the spec).
+ */
+ if (http_auth.header_is_last_match && isspace(*buf.buf)) {
+ /* Trim leading whitespace from this continuation hdr line. */
+ strbuf_ltrim(&buf);
+
+ /*
+ * At this point we should always have at least one existing
+ * value, even if it is empty. Do not bother appending the new
+ * value if this continuation header is itself empty.
+ */
+ if (!values->nr) {
+ BUG("should have at least one existing header value");
+ } else if (buf.len) {
+ char *prev = xstrdup(values->v[values->nr - 1]);
+
+ /* Join two non-empty values with a single space. */
+ const char *const sp = *prev ? " " : "";
+
+ strvec_pop(values);
+ strvec_pushf(values, "%s%s%s", prev, sp, buf.buf);
+ free(prev);
+ }
+
+ goto exit;
+ }
+
+ /* This is the start of a new header we don't care about */
+ http_auth.header_is_last_match = 0;
+
+ /*
+ * If this is a HTTP status line and not a header field, this signals
+ * a different HTTP response. libcurl writes all the output of all
+ * response headers of all responses, including redirects.
+ * We only care about the last HTTP request response's headers so clear
+ * the existing array.
+ */
+ if (istarts_with(buf.buf, "http/"))
+ strvec_clear(values);
+
+exit:
+ strbuf_release(&buf);
+ return size;
+}
+
size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)
{
return nmemb;
@@ -1864,6 +1956,8 @@ static int http_request(const char *url,
fwrite_buffer);
}
+ curl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);
+
accept_language = http_get_accept_language_header();
if (accept_language)
--
gitgitgadget
^ permalink raw reply related
* [PATCH v7 10/12] http: replace unsafe size_t multiplication with st_mult
From: Matthew John Cheetham via GitGitGadget @ 2023-01-20 22:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Lessley Dennington, Matthew John Cheetham,
M Hickford, Jeff Hostetler, Glen Choo, Victoria Dye,
Ævar Arnfjörð Bjarmason, Matthew John Cheetham,
Matthew John Cheetham
In-Reply-To: <pull.1352.v7.git.1674252530.gitgitgadget@gmail.com>
From: Matthew John Cheetham <mjcheetham@outlook.com>
Replace direct multiplication of two size_t parameters in curl response
stream handling callback functions with `st_mult` to guard against
overflows.
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
---
http.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/http.c b/http.c
index 8a5ba3f4776..a2a80318bb2 100644
--- a/http.c
+++ b/http.c
@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;
size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
{
- size_t size = eltsize * nmemb;
+ size_t size = st_mult(eltsize, nmemb);
struct buffer *buffer = buffer_;
if (size > buffer->buf.len - buffer->posn)
@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)
size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)
{
- size_t size = eltsize * nmemb;
+ size_t size = st_mult(eltsize, nmemb);
struct strbuf *buffer = buffer_;
strbuf_add(buffer, ptr, size);
--
gitgitgadget
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox