GCC Compile farm (Linux, Solaris, AIX etc.) testing of git.git

GCC Compile farm (Linux, Solaris, AIX etc.) testing of git.git

[Ævar Arnfjörð Bjarmason]

I've had access to the GCC Compile Farm for testing on various
architectures for a while. Around the 2.19.0 release I submitted some
patches / bug reports found there.

I've now improved this to run it via GitLab CI & made the output
accessible. Outline of how this works at
https://gitlab.com/git-vcs/git-gitlab-ci/commit/497805b18f

https://gitlab.com/git-vcs/git-ci/branches is a repo that houses a merge
of {master,next,pu} from git.git and that git-gitlab-ci.git repo.

There's still plenty of rough edges to it, but
https://gitlab.com/git-vcs/git-ci/-/jobs already has some useful
results.

Right now this is all pushed out manually. But I'm planning to get it to
a point where soon after Junio pushes changes out a run will kick off on
all these platforms.

The machines it's running on & their build / test configuration is also
something I set up as a one-off.  The full list of available machines is
at https://cfarm.tetaneutral.net/machines/list/ and I picked some subset
that looked interesting (outlier platforms) at
https://gitlab.com/git-vcs/git-gitlab-ci/blob/master/.gitlab-ci.yml

Missed Commit in 2.20.1

[Randall S. Becker]

On April 2, 2018 4:02 PM, Stefan Beller found my change:
> On Mon, Apr 2, 2018 at 12:57 PM, Randall S. Becker
> <rsbecker@nexbridge.com> wrote:
> > On April 2, 2018 3:34 PM, Junio C Hamano wrote:
> >> The latest feature release Git v2.17.0 is now available at the usual
> >> places.  It is comprised of 516 non-merge commits since v2.16.0,
> >> contributed by 71 people, 20 of which are new faces.
> >
> > Just a heads up. I think this one might have gotten missed at some point
a
> few months back. I think it was submitted back in January. Not sure where
it
> fell off or whether it was just dropped.
> >
> > diff --git a/transport-helper.c b/transport-helper.c index
> > 3f380d87d..5ee7007f6 100644
> > --- a/transport-helper.c
> > +++ b/transport-helper.c
> > @@ -1212,7 +1212,7 @@ static int udt_do_read(struct
> unidirectional_transfer *t)
> >                 return 0;       /* No space for more. */
> >
> >         transfer_debug("%s is readable", t->src_name);
> > -       bytes = read(t->src, t->buf + t->bufuse, BUFFERSIZE -
t->bufuse);
> > +       bytes = xread(t->src, t->buf + t->bufuse, BUFFERSIZE -
t->bufuse);
> >         if (bytes < 0 && errno != EWOULDBLOCK && errno != EAGAIN &&
> >                 errno != EINTR) {
> >                 error_errno("read(%s) failed", t->src_name);
> 
> Patch at https://public-
> inbox.org/git/010f01d38a9e$a5c4f290$f14ed7b0$@nexbridge.com/

Just a subtle reminder that this particular change does not seem to have
made it into 2.20.1. I've retrofitted xread into the NonStop port and will
be re-submitting it with the remaining platform changes assuming the test
suite runs at least as well as our last port at 2.17.0 (in progress as we
speak). We are down to four variances from standard code:

*    git-compat-util.h referencing FLOSS headers needed on platform.
*    this one (xread instead of read in transport-helper.c)
*    config.mak.uname specific settings for NonStop not currently present.
*    t/Makefile - have to comment out all test-lint dependencies because of
segfault at 2.20.1. (to be investigated but we don't submit this unless we
find a good fix)

Please stay tuned for patches. We are very much looking forward to having
the two (or three) different NonStop hardware personalities supported
without mods in the very near future. Our goal, assuming those patches are
acceptable, is to move our build/test/distro into a Jenkins config that runs
with minimal human involvement (a.k.a. me).

Cheers,
Randall

-- Brief whoami:
  NonStop developer since approximately NonStop(211288444200000000)
  UNIX developer since approximately 421664400
-- In my real life, I talk too much.

Re: Missed Commit in 2.20.1

[Ævar Arnfjörð Bjarmason]

On Wed, Dec 26 2018, Randall S. Becker wrote:

> Please stay tuned for patches. We are very much looking forward to having
> the two (or three) different NonStop hardware personalities supported
> without mods in the very near future. Our goal, assuming those patches are
> acceptable, is to move our build/test/distro into a Jenkins config that runs
> with minimal human involvement (a.k.a. me).

Portability patches like that are definitely wanted.

In case you haven't seen my recent work on getting GitLab CI up &
running check out
https://public-inbox.org/git/875zwm15k2.fsf@evledraar.gmail.com/

It differs from existing CI implementations for git.git in being focused
on doing the actual run on remote hosts that can be ssh'd to.

So perhaps you'd be interested in some of:

a) Contributing a NonStop box to the GCC Compile Farm
   (https://cfarm.tetaneutral.net/machines/list/). Then I can add it to
   my tests, but also other people porting free software will fix bugs
   pro-actively before you spot them.

b) I now have a gitlab-runner I maintain powering this git-ci.git stuff
   & presenting it on gitlab.com, if you give me SSH access I can add it
   to my own runs...

c) ...or you can just run your own gitlab-runner on
   https://gitlab.com/git-vcs/git-ci/ (although this amounts to giving
   me ssh access, since you'll be running my code)....

d) ... or reuse the CI code I wrote to setup your own runner/pusher
   against NonStop, only you'd have access to this....

e) Or do whatever you're planning with Jenkins.

If you want to just go with e) that's fine, just saying that you could
re-use some existing stuff with a-d) if you wanted.

RE: Missed Commit in 2.20.1

[Randall S. Becker]

On December 27, 2018 17:40, Ævar Arnfjörð Bjarmason wrote:
> On Wed, Dec 26 2018, Randall S. Becker wrote:
> 
> > Please stay tuned for patches. We are very much looking forward to
> > having the two (or three) different NonStop hardware personalities
> > supported without mods in the very near future. Our goal, assuming
> > those patches are acceptable, is to move our build/test/distro into a
> > Jenkins config that runs with minimal human involvement (a.k.a. me).
> 
> Portability patches like that are definitely wanted.
> 
> In case you haven't seen my recent work on getting GitLab CI up & running
> check out https://public-
> inbox.org/git/875zwm15k2.fsf@evledraar.gmail.com/
> 
> It differs from existing CI implementations for git.git in being focused on
> doing the actual run on remote hosts that can be ssh'd to.
> 
> So perhaps you'd be interested in some of:
> 
> a) Contributing a NonStop box to the GCC Compile Farm
>    (https://cfarm.tetaneutral.net/machines/list/). Then I can add it to
>    my tests, but also other people porting free software will fix bugs
>    pro-actively before you spot them.

If I win the lottery, sure. Right now, a contribution like that is a bit beyond my budget. I'm not sure that anything "GCC" will fly with management since GCC does not port to the platform at all at this point in time. Many have tried. Many have failed. We're limited to c89 and c99.

> b) I now have a gitlab-runner I maintain powering this git-ci.git stuff
>    & presenting it on gitlab.com, if you give me SSH access I can add it
>    to my own runs...

Sorry, no can do on this one.

> c) ...or you can just run your own gitlab-runner on
>    https://gitlab.com/git-vcs/git-ci/ (although this amounts to giving
>    me ssh access, since you'll be running my code)....

This may be more possible. I've been considering putting up a GitLab instance but it's a matter of not having enough time. I have more than enough LXC ubuntu instances still available for something like that.

> d) ... or reuse the CI code I wrote to setup your own runner/pusher
>    against NonStop, only you'd have access to this....

More likely. Private chat worth it perhaps.

> e) Or do whatever you're planning with Jenkins.

We are currently using Jenkins to build/test git. I was thinking about contributing a Jenkinsfile that would build on a Controller (what happens today for our git port), or setting up a parameterized form for SSH for an Agent that might be better in a farm setting. I am close to the point where human interaction is limited to 'git branch -f production vn.mm.l' and git is tested and built for distribution without further touching. At least once my platform patches are applied it will be.

> If you want to just go with e) that's fine, just saying that you could re-use
> some existing stuff with a-d) if you wanted.

I am interested. Let's see how my $DAYJOB goes in the next few months. I really do like the idea of setting up a community instance of GitLab to do this and include a CI runner. Hmmm.

Cheers,
Randall

Re: Missed Commit in 2.20.1

[Ævar Arnfjörð Bjarmason]

On Thu, Dec 27 2018, Randall S. Becker wrote:

> On December 27, 2018 17:40, Ævar Arnfjörð Bjarmason wrote:
>> On Wed, Dec 26 2018, Randall S. Becker wrote:
>>
>> > Please stay tuned for patches. We are very much looking forward to
>> > having the two (or three) different NonStop hardware personalities
>> > supported without mods in the very near future. Our goal, assuming
>> > those patches are acceptable, is to move our build/test/distro into a
>> > Jenkins config that runs with minimal human involvement (a.k.a. me).
>>
>> Portability patches like that are definitely wanted.
>>
>> In case you haven't seen my recent work on getting GitLab CI up & running
>> check out https://public-
>> inbox.org/git/875zwm15k2.fsf@evledraar.gmail.com/
>>
>> It differs from existing CI implementations for git.git in being focused on
>> doing the actual run on remote hosts that can be ssh'd to.
>>
>> So perhaps you'd be interested in some of:
>>
>> a) Contributing a NonStop box to the GCC Compile Farm
>>    (https://cfarm.tetaneutral.net/machines/list/). Then I can add it to
>>    my tests, but also other people porting free software will fix bugs
>>    pro-actively before you spot them.
>
> If I win the lottery, sure. Right now, a contribution like that is a bit beyond my budget. I'm not sure that anything "GCC" will fly with management since GCC does not port to the platform at all at this point in time. Many have tried. Many have failed. We're limited to c89 and c99.
>
>> b) I now have a gitlab-runner I maintain powering this git-ci.git stuff
>>    & presenting it on gitlab.com, if you give me SSH access I can add it
>>    to my own runs...
>
> Sorry, no can do on this one.
>
>> c) ...or you can just run your own gitlab-runner on
>>    https://gitlab.com/git-vcs/git-ci/ (although this amounts to giving
>>    me ssh access, since you'll be running my code)....
>
> This may be more possible. I've been considering putting up a GitLab instance but it's a matter of not having enough time. I have more than enough LXC ubuntu instances still available for something like that.

FWIW it's gitlab-runner, not gitlab, you can run gitlab-runner (and I
do) without installing any of the rest of gitlab. It's basically a
daemon that sits in a loop polling to see if there's new jobs for it,

So the extent of the setup is that I have a Debian box that has:

    vm ~ (master=) $ sudo grep -v token /etc/gitlab-runner/config.toml
    concurrent = 10
    check_interval = 30

    [[runners]]
      name = "gcc-farm"
      url = "https://gitlab.com/"
      executor = "shell"
      [runners.cache]

(Secret token pruned out), that's what sets it up as "gcc-farm" (see
https://gitlab.com/git-vcs/git-ci/blob/master/.gitlab-ci.yml), which
just runs this shellscript:
https://gitlab.com/git-vcs/git-ci/blob/master/ci/gitlab/run-on-gcc-farm.sh

I.e. for a given job name (it extracts the hostname from that) it ssh's
to that machine after scp-ing a given git.git revision to it, compiles &
tests, and all the output is then visible at :
https://gitlab.com/git-vcs/git-ci/-/jobs?scope=finished

I still need to write the rather trivial bit that'll run this as a
cronjob and push as new git.git revs become available, but so far I've
been wanting to get it passing 100% as a baseline, which hasn't happend
due to wanting to handle transitory failures (e.g. ssh to some boxes
timing out) and smoking out the various intermittent failures on some of
the odd platforms/distro versions.

>> d) ... or reuse the CI code I wrote to setup your own runner/pusher
>>    against NonStop, only you'd have access to this....
>
> More likely. Private chat worth it perhaps.

Sure, any time.

>> e) Or do whatever you're planning with Jenkins.
>
> We are currently using Jenkins to build/test git. I was thinking about contributing a Jenkinsfile that would build on a Controller (what happens today for our git port), or setting up a parameterized form for SSH for an Agent that might be better in a farm setting. I am close to the point where human interaction is limited to 'git branch -f production vn.mm.l' and git is tested and built for distribution without further touching. At least once my platform patches are applied it will be.

Makes sense, I thought you were just working on this as a new thing to
do CI testing.

>> If you want to just go with e) that's fine, just saying that you could re-use
>> some existing stuff with a-d) if you wanted.
>
> I am interested. Let's see how my $DAYJOB goes in the next few months. I really do like the idea of setting up a community instance of GitLab to do this and include a CI runner. Hmmm.

As noted above it's just a runner that's needed. I'm using gitlab.com's
instance to present the results, I *could* also setup my own, but no
reason to if they're willing to host it for free.

B.t.w. I'm sure the same can be done with GitHub/Travis or Azure CI
etc., I was just familiar with GitLab's, and wanted something where to
begin with I could have my own overlay of custom CI on top of git.git
without integrating patches upstream yet while I see if that even makes
sense.