RE: t5570 - not cloned error

RE: t5570 - not cloned error

[Randall S. Becker]

Sorry to repost - ended up in my own spam trap.

On May 1, 2015 11:05 AM, I wrote, in my haste:
> 
> Greetings - and asking for a bit of help resolving test failures.
> 
> I'm having an issue with t5570 at 2.3.7 which seems to be a regression
from
> 2.3.3 (currently installed), but I cannot be sure. This test failed prior
to
> 2.3.0 in the box, worked from 2.3.0 to 2.3.3 - suggesting that it may be
> environmental, not actually in git. Making some assumptions, it looks like
> the URL for the test repository is not correct and may depend on localhost
> resolving properly - which DNS does not do well on this box (outside my
> control, we are multi-home, and localhost does not resolve to 127.0.0.1 or
> [::1]). Only t5570 #'s 3-5 fail and I found a strange message in the
output
> of the test seemingly referring to a bad repo name. I would really
> appreciate some pointers on where to look next and how to go about
resolving
> this. I am happy to try to work through this on 2.4.0 if that would be
more
> efficient for the team. Anything relating to git-daemon makes me nervous
in
> terms of installing the code.
> 
> Platform is HP NonStop (Posix-esque environment):
> 
> In the test output:
> *** t5570-git-daemon.sh ***
> <snip>
> not ok 3 - clone git repository
> #
> #               git clone "$GIT_DAEMON_URL/repo.git" clone &&
> #               test_cmp file clone/file
> #
> not ok 4 - fetch changes via git protocol
> #
> #               echo content >>file &&
> #               git commit -a -m two &&
> #               git push public &&
> #               (cd clone && git pull) &&
> #               test_cmp file clone/file
> #
> not ok 5 - remote detects correct HEAD
> #
> #               git push public master:other &&
> #               (cd clone &&
> #                git remote set-head -d origin &&
> #                git remote set-head -a origin &&
> 
> And
> 
> ../git/t/trash directory.t5570-git-daemon: cat output
> fatal: remote error: repository not exported: /repo.git
> 
> Additional context: t0025, t0301, t3900, t9001, t9020 are not 100% but the
> issues are acceptable - we can discuss separately.

We definitely have an issue with localhost. When forcing the DNS resolver to
return 127.0.0.1, we pass 1-16 then 17 fails as I expected to happen based
on my DNS futzing. Heads up that this test is not-surprisingly sensitive to
DNS problems. My environment is still in a messy state where I can reproduce
the original problem so it might be a useful moment for me to find a way to
modify the test script to harden it. Any suggestion on that score (as in
where and
roughly how it might be made more reliable)?

Note: Since the original post, I moved the fork to 2.4.0.

Cheers,
Randall

Re: t5570 - not cloned error

[Junio C Hamano]

"Randall S. Becker" <rsbecker@nexbridge.com> writes:

> We definitely have an issue with localhost. When forcing the DNS resolver to
> return 127.0.0.1, we pass 1-16 then 17 fails as I expected to happen based
> on my DNS futzing. Heads up that this test is not-surprisingly sensitive to
> DNS problems. My environment is still in a messy state where I can reproduce
> the original problem so it might be a useful moment for me to find a way to
> modify the test script to harden it. Any suggestion on that score
> (as in where and roughly how it might be made more reliable)?

I do not think this counts as a useful "suggestion", but is this
"resolver does not work for local as expected" case even worth
protecting our tests against?

RE: t5570 - not cloned error

[Randall S. Becker]

On May 5, 2015 6:01 PM Junio C Hamano wrote:
> "Randall S. Becker" <rsbecker@nexbridge.com> writes:
> > We definitely have an issue with localhost. When forcing the DNS
resolver to
> > return 127.0.0.1, we pass 1-16 then 17 fails as I expected to happen
based
> > on my DNS futzing. Heads up that this test is not-surprisingly sensitive
to
> > DNS problems. My environment is still in a messy state where I can
reproduce
> > the original problem so it might be a useful moment for me to find a way
to
> > modify the test script to harden it. Any suggestion on that score
> > (as in where and roughly how it might be made more reliable)?
> 
> I do not think this counts as a useful "suggestion", but is this
> "resolver does not work for local as expected" case even worth
> protecting our tests against?

I see your point, but after having spent "way too much time" away from the
$DAYJOB tracking this down, I was hoping to catch the root cause earlier
next time. Perhaps adding a test step validating that localhost comes back
with a reasonable value - whatever that may be in context. I'm just not sure
what the test really needs at its heart to run properly - obviously the IP
address of the system as  visible in our DMZ is not working for the test. 

Re: t5570 - not cloned error

[Jeff King]

On Tue, May 05, 2015 at 06:23:41PM -0400, Randall S. Becker wrote:

> 
> On May 5, 2015 6:01 PM Junio C Hamano wrote:
> > "Randall S. Becker" <rsbecker@nexbridge.com> writes:
> > > We definitely have an issue with localhost. When forcing the DNS
> resolver to
> > > return 127.0.0.1, we pass 1-16 then 17 fails as I expected to happen
> based
> > > on my DNS futzing. Heads up that this test is not-surprisingly sensitive
> to
> > > DNS problems. My environment is still in a messy state where I can
> reproduce
> > > the original problem so it might be a useful moment for me to find a way
> to
> > > modify the test script to harden it. Any suggestion on that score
> > > (as in where and roughly how it might be made more reliable)?
> > 
> > I do not think this counts as a useful "suggestion", but is this
> > "resolver does not work for local as expected" case even worth
> > protecting our tests against?
> 
> I see your point, but after having spent "way too much time" away from the
> $DAYJOB tracking this down, I was hoping to catch the root cause earlier
> next time. Perhaps adding a test step validating that localhost comes back
> with a reasonable value - whatever that may be in context. I'm just not sure
> what the test really needs at its heart to run properly - obviously the IP
> address of the system as  visible in our DMZ is not working for the test.

I'm having trouble even understanding who is looking up "localhost"
here.  All of the git-daemon tests should be directly using 127.0.0.1 in
the URLs (i.e., what is in $GIT_DAEMON_URL). If that is a problem, I
think you might be able to parameterize the way we set up
$GIT_DAEMON_URL (and how we invoked git-daemon; see lib-git-daemon.sh).

In the interpolation tests, we do use the string "localhost" but we
should never do a lookup on it. We set an environment variable that
tells the git client to _tell_ the server we looked up localhost, but we
should still be accessing it as 127.0.0.1.

So I'm confused about what the actual problem is, or why the test cares
about resolving "localhost" in the first place.

-Peff

RE: t5570 - not cloned error

[Randall S. Becker]

On May 5, 2015 6:55 PM Jeff King wrote:
> On Tue, May 05, 2015 at 06:23:41PM -0400, Randall S. Becker wrote:
> > On May 5, 2015 6:01 PM Junio C Hamano wrote:
> > > "Randall S. Becker" <rsbecker@nexbridge.com> writes:
> > > > We definitely have an issue with localhost. When forcing the DNS
> > resolver to
> > > > return 127.0.0.1, we pass 1-16 then 17 fails as I expected to happen
> > based
> > > > on my DNS futzing. Heads up that this test is not-surprisingly sensitive
> > to
> > > > DNS problems. My environment is still in a messy state where I can
> > reproduce
> > > > the original problem so it might be a useful moment for me to find a way
> > to
> > > > modify the test script to harden it. Any suggestion on that score
> > > > (as in where and roughly how it might be made more reliable)?
> > >
> > > I do not think this counts as a useful "suggestion", but is this
> > > "resolver does not work for local as expected" case even worth
> > > protecting our tests against?
> >
> > I see your point, but after having spent "way too much time" away from the
> > $DAYJOB tracking this down, I was hoping to catch the root cause earlier
> > next time. Perhaps adding a test step validating that localhost comes back
> > with a reasonable value - whatever that may be in context. I'm just not sure
> > what the test really needs at its heart to run properly - obviously the IP
> > address of the system as  visible in our DMZ is not working for the test.
> 
> I'm having trouble even understanding who is looking up "localhost"
> here.  All of the git-daemon tests should be directly using 127.0.0.1 in
> the URLs (i.e., what is in $GIT_DAEMON_URL). If that is a problem, I
> think you might be able to parameterize the way we set up
> $GIT_DAEMON_URL (and how we invoked git-daemon; see lib-git-daemon.sh).
> 
> In the interpolation tests, we do use the string "localhost" but we
> should never do a lookup on it. We set an environment variable that
> tells the git client to _tell_ the server we looked up localhost, but we
> should still be accessing it as 127.0.0.1.
> 
> So I'm confused about what the actual problem is, or why the test cares
> about resolving "localhost" in the first place.

That gives me a pretty good idea of where to start looking. I will first hunt down the resolution and go from there. Stay tuned - it may be a day or two before I have a chance to do more than a cursory debug.

Cheers,
Randall

RE: t5570 - not cloned error

[Randall S. Becker]

On May 5, 2015 8:46 PM, I wrote in my haste:
> On May 5, 2015 6:55 PM Jeff King wrote:
> > On Tue, May 05, 2015 at 06:23:41PM -0400, Randall S. Becker wrote:
> > > On May 5, 2015 6:01 PM Junio C Hamano wrote:
> > > > "Randall S. Becker" <rsbecker@nexbridge.com> writes:
> > > > > We definitely have an issue with localhost. When forcing the DNS
> > > resolver to
> > > > > return 127.0.0.1, we pass 1-16 then 17 fails as I expected to happen
> > > based
> > > > > on my DNS futzing. Heads up that this test is not-surprisingly sensitive
> > > to
> > > > > DNS problems. My environment is still in a messy state where I can
> > > reproduce
> > > > > the original problem so it might be a useful moment for me to find a way
> > > to
> > > > > modify the test script to harden it. Any suggestion on that score
> > > > > (as in where and roughly how it might be made more reliable)?
> > > >
> > > > I do not think this counts as a useful "suggestion", but is this
> > > > "resolver does not work for local as expected" case even worth
> > > > protecting our tests against?
> > >
> > > I see your point, but after having spent "way too much time" away from the
> > > $DAYJOB tracking this down, I was hoping to catch the root cause earlier
> > > next time. Perhaps adding a test step validating that localhost comes back
> > > with a reasonable value - whatever that may be in context. I'm just not sure
> > > what the test really needs at its heart to run properly - obviously the IP
> > > address of the system as  visible in our DMZ is not working for the test.
> >
> > I'm having trouble even understanding who is looking up "localhost"
> > here.  All of the git-daemon tests should be directly using 127.0.0.1 in
> > the URLs (i.e., what is in $GIT_DAEMON_URL). If that is a problem, I
> > think you might be able to parameterize the way we set up
> > $GIT_DAEMON_URL (and how we invoked git-daemon; see lib-git-
> daemon.sh).
> >
> > In the interpolation tests, we do use the string "localhost" but we
> > should never do a lookup on it. We set an environment variable that
> > tells the git client to _tell_ the server we looked up localhost, but we
> > should still be accessing it as 127.0.0.1.
> >
> > So I'm confused about what the actual problem is, or why the test cares
> > about resolving "localhost" in the first place.
> 
> That gives me a pretty good idea of where to start looking. I will first hunt
> down the resolution and go from there. Stay tuned - it may be a day or two
> before I have a chance to do more than a cursory debug.

I found the root cause, being that the multi-IP stack configuration on this platform is not being passed correctly through bash to the shell that make starts when running tests. When running the test under ksh, which has a deeper port, the test behaves correctly. As it turns out, this is a platform-specific issue with our bash port.

This is not git's issue at all, nor the DNS resolver configuration (although I did find an independent problem there). Sorry to be a bother.

Cheers,
Randall