RE[3]: Distribute continuous integration (build) to servers outside docker container

["dhinakar.k" <dhinakar.k@samsung.com>]

Hi Tim,

Thanks a lot for your response.
Please find below the details that you requested.

# du -sh /var/lib/Jenkins
32G     /var/lib/Jenkins

# du -sh /var/lib/jenkins/* | sort -h
0       /var/lib/jenkins/secret.key.not-so-secret
4.0K    /var/lib/jenkins/com.cloudbees.hudson.plugins.folder.config.AbstractFolderConfiguration.xml
4.0K    /var/lib/jenkins/com.mtvi.plateng.hudson.ldap.LdapMailAddressResolver.xml
4.0K    /var/lib/jenkins/com.orctom.jenkins.plugin.buildtimestamp.BuildTimestampWrapper.xml
4.0K    /var/lib/jenkins/credentials.xml
4.0K    /var/lib/jenkins/hudson.maven.MavenModuleSet.xml
4.0K    /var/lib/jenkins/hudson.model.UpdateCenter.xml
4.0K    /var/lib/jenkins/hudson.plugins.emailext.ExtendedEmailPublisher.xml
4.0K    /var/lib/jenkins/hudson.plugins.git.GitSCM.xml
4.0K    /var/lib/jenkins/hudson.plugins.git.GitTool.xml
4.0K    /var/lib/jenkins/hudson.plugins.sidebar_link.SidebarLinkPlugin.xml
4.0K    /var/lib/jenkins/hudson.plugins.sonar.SonarGlobalConfiguration.xml
4.0K    /var/lib/jenkins/hudson.plugins.timestamper.TimestamperConfig.xml
4.0K    /var/lib/jenkins/hudson.tasks.Mailer.xml
4.0K    /var/lib/jenkins/hudson.tasks.Shell.xml
4.0K    /var/lib/jenkins/hudson.triggers.SCMTrigger.xml
4.0K    /var/lib/jenkins/identity.key.enc
4.0K    /var/lib/jenkins/io.jenkins.plugins.junit.storage.JunitTestResultStorageConfiguration.xml
4.0K    /var/lib/jenkins/jenkins.fingerprints.GlobalFingerprintConfiguration.xml
4.0K    /var/lib/jenkins/jenkins.install.InstallUtil.lastExecVersion
4.0K    /var/lib/jenkins/jenkins.install.UpgradeWizard.state
4.0K    /var/lib/jenkins/jenkins.model.ArtifactManagerConfiguration.xml
4.0K    /var/lib/jenkins/jenkins.model.GlobalBuildDiscarderConfiguration.xml
4.0K    /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml
4.0K    /var/lib/jenkins/jenkins.plugins.msginject.MsgInjectConfig.xml
4.0K    /var/lib/jenkins/jenkins.security.ResourceDomainConfiguration.xml
4.0K    /var/lib/jenkins/jenkins.security.apitoken.ApiTokenPropertyConfiguration.xml
4.0K    /var/lib/jenkins/jenkins.tasks.filters.EnvVarsFilterGlobalConfiguration.xml
4.0K    /var/lib/jenkins/jenkins.telemetry.Correlator.xml
4.0K    /var/lib/jenkins/net.plavcak.jenkins.plugins.scmskip.SCMSkipBuildWrapper.xml
4.0K    /var/lib/jenkins/net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.xml
4.0K    /var/lib/jenkins/nodeMonitors.xml
4.0K    /var/lib/jenkins/org.codefirst.SimpleThemeDecorator.xml
4.0K    /var/lib/jenkins/org.jenkinsCi.plugins.projectDescriptionSetter.DescriptionSetterWrapper.xml
4.0K    /var/lib/jenkins/org.jenkinsci.plugins.ansible_tower.AnsibleTower.xml
4.0K    /var/lib/jenkins/org.jenkinsci.plugins.corsfilter.AccessControlsFilter.xml
4.0K    /var/lib/jenkins/org.jenkinsci.plugins.lucene.search.config.SearchBackendConfiguration.xml
4.0K    /var/lib/jenkins/org.jenkinsci.plugins.workflow.flow.GlobalDefaultFlowDurabilityLevel.xml
4.0K    /var/lib/jenkins/org.quality.gates.jenkins.plugin.GlobalConfig.xml
4.0K    /var/lib/jenkins/queue.xml
4.0K    /var/lib/jenkins/queue.xml.bak
4.0K    /var/lib/jenkins/scriptApproval.xml
4.0K    /var/lib/jenkins/secret.key
4.0K    /var/lib/jenkins/sidebar-link.xml
12K     /var/lib/jenkins/gerrit-trigger.xml
16K     /var/lib/jenkins/hudson.plugins.ansicolor.AnsiColorBuildWrapper.xml
20K     /var/lib/jenkins/config.xml
44K     /var/lib/jenkins/nodes
72K     /var/lib/jenkins/secrets
356K    /var/lib/jenkins/users
1.7M    /var/lib/jenkins/logs
1.8M    /var/lib/jenkins/luceneIndex
3.6M    /var/lib/jenkins/updates
3.9M    /var/lib/jenkins/plugins_backup
316M    /var/lib/jenkins/plugins
316M    /var/lib/jenkins/plugins_not_working
372M    /var/lib/jenkins/plugins_old_2ndFeb2024
376M    /var/lib/jenkins/plugins_working_2ndFeb2024
1.8G    /var/lib/jenkins/userContent
11G     /var/lib/jenkins/jobs
19G     /var/lib/jenkins/workspace

# du -sh /fuego-rw
13G     /fuego-rw

# du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Dhrystone-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-aarch64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.IOzone-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Interbench-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Java-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.OpenSSL-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Stream-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Whetstone-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.bonnie-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.cyclictest-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.dbench4-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ebizzy-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ffsb-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.fio-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.gtkperf-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.hackbench-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.himeno-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.iperf-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.linpack-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.lmbench2-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.nbench_byte-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netperf-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netpipe-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.signaltest-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.tiobench-x86_64
0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.x11perf-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Dhrystone-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.GLMark-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.IOzone-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Interbench-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Java-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.OpenSSL-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Stream-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Whetstone-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.bonnie-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.cyclictest-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.dbench4-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ebizzy-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ffsb-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.fio-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.gtkperf-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.hackbench-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.himeno-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.iperf-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.linpack-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.lmbench2-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.nbench_byte-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netperf-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netpipe-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.signaltest-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.tiobench-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.x11perf-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.LTP-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.OpenSSL-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.aiostress-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bc-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bzip2-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.crashme-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.expat-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.fontconfig-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ft2demos-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.glib-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.hello_world-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ipv6connect-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.jpeg-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.linus_stress-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.netperf-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.pi_tests-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.rmaptest-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.scrashme-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.stress-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.synctest-x86_64
0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.zlib-x86_64
4.0K    /fuego-rw/buildzone/README
12K     /fuego-rw/buildzone/Functional.bc-x86_64
32K     /fuego-rw/buildzone/Functional.hello_world-x86_64
32K     /fuego-rw/buildzone/Functional.linus_stress-x86_64
32K     /fuego-rw/buildzone/Functional.synctest-x86_64
36K     /fuego-rw/buildzone/Functional.rmaptest-x86_64
48K     /fuego-rw/buildzone/Benchmark.himeno-x86_64
48K     /fuego-rw/buildzone/Benchmark.linpack-x86_64
48K     /fuego-rw/buildzone/Functional.ipv6connect-x86_64
60K     /fuego-rw/buildzone/Benchmark.Stream-x86_64
76K     /fuego-rw/buildzone/Benchmark.Whetstone-x86_64
76K     /fuego-rw/buildzone/Benchmark.ebizzy-x86_64
84K     /fuego-rw/buildzone/Functional.aiostress-x86_64
168K    /fuego-rw/buildzone/Benchmark.Dhrystone-x86_64
240K    /fuego-rw/buildzone/Benchmark.tiobench-x86_64
244K    /fuego-rw/buildzone/Functional.crashme-x86_64
488K    /fuego-rw/buildzone/Benchmark.Interbench-x86_64
576K    /fuego-rw/buildzone/Functional.scrashme-x86_64
760K    /fuego-rw/buildzone/Benchmark.hackbench-x86_64
816K    /fuego-rw/buildzone/Benchmark.signaltest-x86_64
868K    /fuego-rw/buildzone/Functional.pi_tests-x86_64
900K    /fuego-rw/buildzone/Benchmark.netpipe-x86_64
904K    /fuego-rw/buildzone/Benchmark.cyclictest-x86_64
1004K   /fuego-rw/buildzone/Functional.stress-x86_64
1.3M    /fuego-rw/buildzone/Benchmark.ffsb-x86_64
1.6M    /fuego-rw/buildzone/Benchmark.GLMark-aarch64
1.6M    /fuego-rw/buildzone/Benchmark.GLMark-x86_64
1.7M    /fuego-rw/buildzone/Benchmark.nbench_byte-x86_64
2.1M    /fuego-rw/buildzone/Benchmark.iperf-x86_64
2.3M    /fuego-rw/buildzone/Benchmark.gtkperf-x86_64
2.4M    /fuego-rw/buildzone/Benchmark.bonnie-x86_64
2.5M    /fuego-rw/buildzone/Benchmark.x11perf-x86_64
2.9M    /fuego-rw/buildzone/Functional.bzip2-x86_64
3.0M    /fuego-rw/buildzone/Functional.zlib-x86_64
3.1M    /fuego-rw/buildzone/Benchmark.IOzone-x86_64
3.5M    /fuego-rw/buildzone/Functional.jpeg-x86_64
5.2M    /fuego-rw/buildzone/Benchmark.lmbench2-x86_64
6.1M    /fuego-rw/buildzone/Benchmark.netperf-x86_64
6.1M    /fuego-rw/buildzone/Functional.netperf-x86_64
7.0M    /fuego-rw/buildzone/Functional.fontconfig-x86_64
9.1M    /fuego-rw/buildzone/Benchmark.fio-x86_64
29M     /fuego-rw/buildzone/Functional.expat-x86_64
33M     /fuego-rw/buildzone/Benchmark.dbench4-x86_64
34M     /fuego-rw/buildzone/Functional.ft2demos-x86_64
49M     /fuego-rw/buildzone/Functional.glib-x86_64
74M     /fuego-rw/buildzone/Benchmark.OpenSSL-x86_64
74M     /fuego-rw/buildzone/Functional.OpenSSL-x86_64
160M    /fuego-rw/buildzone/Benchmark.Java-x86_64
1.8G    /fuego-rw/buildzone/X86-SSH-Test.default.Functional.kernel_build-x86_64
2.4G    /fuego-rw/buildzone/Functional.LTP-x86_64

Outside the container, can you run:
$ docker image ls
nxo-container-backup-2ndsep2021          latest         b6a3d16508e8   2 years ago     173GB
fuego-base-image-20thaug2019-container   latest         b2cf22891ac6   4 years ago     2.41GB
fuegocontainerbackup-4thmar2019          latest         04112c383476   5 years ago     3.34GB
fuego-piper-image-22ndnov2021           latest         0121b1c342e6   6 years ago     10.7GB
fuego-xyz-image-22ndnov2021              latest         0bf6b942db70   6 years ago     5.8GB
fuego-dart-image-22ndnov2021           latest         1d2c6a99dc0a   6 years ago     1.66GB
fuego-pavv-image-22ndnov2021             latest         ba71ede1e872   6 years ago     4.63GB
fuego-gpu-image-22ndnov2021           latest         42718d86dd09   6 years ago     3.41GB

Regarding below request, I just want to lock the Jenkins user interface/ dashboard, so that none can see all the jobs, results, logs and other details without logging in. So to start with even one user with login credentials will help.
But I don't know how Jenkins credentials relate to user accounts.  I haven't ever set up Jenkins to handle multiple user accounts.  Indeed, in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to create or manage new user accounts.

Thanks a lot for all other details, they were very helpful.

Regards,
Dhinakar K.

-----Original Message-----
From: Bird, Tim [mailto:Tim.Bird@sony.com] 
Sent: Wednesday, March 13, 2024 5:47 AM
To: dhinakar.k <dhinakar.k@samsung.com>; fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org
Subject: RE: HTML message rejected: Re: Distribute continuous integration (build) to servers outside docker container

Hey Dhinakar,

Sorry to not respond sooner.  There was a lot to process in your email, and some of it I don't have answers for, but I'll try to give some status and feedback where I can.

> -----Original Message-----
> From: dhinakar.k <dhinakar.k@samsung.com> Hi Tim,
> 
> I had sent a reply to your response on 3rd Feb.2024, not sure if it had reached you.
> I am posting the same message below again.
> 
> Thanks a lot for your prompt reply.
> Iam looking forward to try the updated docker container with an 
> upgraded distribution base from Debian stretch to Debian bullseye, and from a Jenkins version of 2.249.3 to 2.414.1. Please let me whenever it is available.

OK - Will do.  I haven't been able to allocate time for testing and fixing it, so I didn't want to push it to the master branch.  If I pushed something to a testing branch, would you be able to download from there and test it?

One issue that came up is that some of the toolchains for some of my old boards/distros are not supported (ie don't work) in Debian bullseye, and so I was looking for solutions for that.
 
> We are migrating to another server, so Iam looking for best options to 
> transfer all the content to the new server. Since the docker container 
> size is huge I want to trim it down by removing unwanted old logs.

It would be good to know where the container size is coming from.  Is it in the Fuego build directories, the Fuego run directories (where Fuego logs and run artifacts are stored), or in the Jenkins build directories (where Jenkins logs and artifacts are)?  The materials under /fuego-rw should be on the host system, and not inside the container itself (since these are volume mounts).  But I don't know how this space is accounted to the container (if any).  It shouldn't be in any container image.

The materials under /var/lib/jenkins are inside the container image.

Can you run the following commands, inside your container, and send the results:

# du -sh /var/lib/jenkins
# du -sh /var/lib/jenkins/* | sort -h
# du -sh /fuego-rw
# du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l

Outside the container, can you run:
$ docker image ls
(and remove any lines not related to fuego images)

But I don't know how Jenkins credentials relate to user accounts.  I haven't ever set up Jenkins to handle multiple user accounts.  Indeed, in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to create or manage new user accounts.

> Also, I am worried about server crash and hard disk crash due to so many read/write cycles (that happens during continuous integration).
> Hence looking for the best way to back up the container and also to 
> distribute the load & read/write cycles to another server (via an ssh 
> agent). I tried with plain Jenkins via ssh credentials plugin, it seems feasible. Hence if fuego upgrades to debian bullseye it would be really helpful.
OK - I'll keep plugging away at this.

> 
> Also, would like to know, when login credentials support will be 
> available in fuego? Because as a security measure we are supposed to have login credentials for each user. Will be available after debian upgrade?

I believe you are referring to Jenkins credentials?  It appears that Jenkins supports credentials in version 2.414, which is the new version of Jenkins in the new docker container (with the Debian upgrade).  So I believe the answer is yes.

But I don't know how Jenkins credentials relate to user accounts.  I haven't ever set up Jenkins to handle multiple user accounts.  Indeed, in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to create or manage new user accounts.

> With respect to removing the logs I will figure out the best way as I 
> see so many options but not sure if that will reduce the container size directly or need to run some container commands to trim it down. I will explore that as well and email my observations in this forum.

I have considered adding an "rm-run" sub-command to ftc, which would allow you to remove old runs.
However, I'd like to see if it's the run data that is causing the size problem that you are seeing.

If you want to archive the data for an individual run, then you can use 'ftc package-run' and move the resulting package to some long-term storage device (before, ostensibly, removing that run from the system.)

I hope this is helpful.
 -- Tim



> 
> Regards,
> Dhinakar K.
> 
> -----Original Message-----
> From: Bird, Tim [mailto:Tim.Bird@sony.com]
> Sent: Saturday, February 3, 2024 3:44 AM
> To: dhinakar k <dhinakar.k@gmail.com>; fuego+owner@lists.linux.dev; 
> fuego@lists.linuxfoundation.org
> Subject: RE: HTML message rejected: Re: Distribute continuous 
> integration (build) to servers outside docker container
> 
> Dhinakar,  See my answers inline below..
> 
> > -----Original Message-----
> > From: dhinakar k <dhinakar.k@gmail.com> Dear Fuego users,
> >
> > I have a requirement, so I thought of checking with the community if 
> > it was feasible and if so what is the best or most efficient way to 
> > implement it.
> > We have fuego setup in a server and all our continuous integration 
> > jobs (for multiple code bases) are setup in it. But the issue is 
> > there are too many builds happening, which puts pressure on the hard 
> > disk (so many read/writes) and also increases the docker container 
> > size (because of build logs).
> >
> > Hence I would like to know if we can distribute the load to another 
> > server, for e.g. job will be created on our main server but build 
> > will happen on the agent on another server. The idea is to move out 
> > all cpu, memory, storage intensive activity to agents (other 
> > servers) on the network. The resultant logs can also be stored on 
> > those servers (outside docker container) so that docker size won't 
> > be growing rapidly.
> >
> > I tried creating a new node and setup an agent (connect via ssh) but 
> > facing issues in installing the 'SSH Build Agents' plugin because of 
> > compatibility issues (jenkins version needs to be upgraded).
> >
> > If anyone already tried and has a solution for this issue please let me know.
> 
> Unfortunately, I have not tried this, so I'm not familiar with how Jenkins supports distributed jobs.
> 
> But I have a few ideas to share.
> 
> First, I'm working now on an update to the docker container, from a 
> distribution base of Debian stretch to Debian bullseye, and from a 
> Jenkins version of 2.249.3 to 2.414.1 (thanks to a patch provided by Fuego user Yoichi Tachibana).  I have applied the patches, but found some issues with the build instructions for some packages working in Debian bullseye.  So I have not committed the changes to
> the Fuego master branch yet.   I hope to be able to do this soon.
> 
> This might help with the issue of not being able to use the ssh agent plugin for Jenkins.
> 
> Second, it may be possible to do some kind of proxy job submission 
> thing, by creating proxy jobs on the main Jenkins server, but having the jobs actually executed ("built", in Jenkins terminology) on secondary servers.
> 
> There are a number of ways of supporting remote execution, and how you 
> implement it will determine where the CPU cycles and memory usage is 
> during the job, and where the logs end up living (and taking up 
> space.)
> 
> If you are using 'ttc' as your transport layer, recent versions of ttc 
> have support for remote boards via ssh (configured in ttc.conf).  But that would leave logs on the master server.
> 
> There are multiple ways to configure Fuego on the secondary servers - 
> for example using Docker containers or installing Fuego natively.  In any event, it may be possible to set up proxy jobs on the main server that call 'ftc' on the secondary
> servers, to run the jobs of interest.   Depending on how this was structured
> you may end up with build artifacts and logs on the secondary system, 
> and only Jenkins artifacts and logs on the main system (which should save space).
> 
> It might be good to get a more detailed description of your Jenkins setup to be able to understand the options available.
> 
> >
> > Also, what is the best way to backup the container after trimming it 
> > down (remove unwanted files/logs etc. and reduce it's size)?
> 
> I'm not sure how to backup a container in Docker.  There's the 'docker container commit'
> command, but I haven't used that.
> 
> In general, to trim down you data usage you will want to perform a few operations.
> Inside Jenkins, you can delete individual builds.  You can likely 
> automate this (not have to do it manually  through the UI), by using 
> the remote API.  That should work for the material that is inside the container.  Lots of data, however, including the build artifacts and the logs are natively on the host, and are only present in the container through bind mounts.
> 
> This means you can remove this data directly using file operations (e.g. rm) on the host.
> There are a few nuances to this, however, as some of the build directories are referenced via symlinks.
> 
> 'ftc' currently supports removing jobs and nodes.  I don't recall off the top of my head if this includes removing all the builds for a job or not.
> But in any case, it sounds like you only want to remove some of the builds/runs, not the jobs themselves.
> We could possibly support 'ftc rm-run', which could remove specific runs.  'ftc query-runs'
> could be used to generate a list of runs (e.g. that were created before a certain date).
> 
> Please provide a few more details, and maybe we can talk through and/or develop some solutions that will help meet your needs.
> 
>  -- Tim
> 
> >
> > On Thu, Feb 1, 2024 at 11:52 PM <fuego+owner@lists.linux.dev> wrote:
> > >
> > > Greetings!
> > >
> > > This is the mlmmj program managing the <fuego@lists.linux.dev> 
> > > mailing list.
> > >
> > > Your message to <fuego@lists.linux.dev> was not delivered to the 
> > > list because it contained a HTML part. Only text/plain messages 
> > > are allowed on this list.
> > >
> > > Please configure your mail client to only send plain text mail.
> > >
> > > For your reference, the rejected message follows below.
> > >
> > >
> > >
> > > ---------- Forwarded message ----------
> > > From: dhinakar k <dhinakar.k@gmail.com>
> > > To: fuego+help@lists.linux.dev, fuego@lists.linuxfoundation.org, 
> > > "Bird, Timothy" <Tim.Bird@sony.com>
> > > Cc:
> > > Bcc:
> > > Date: Thu, 1 Feb 2024 23:51:30 +0530
> > > Subject: Re: Distribute continuous integration (build) to servers 
> > > outside docker container Dear Fuego users,
> > >
> > > I have a requirement, so I thought of checking with the community 
> > > if it was feasible and if so what is the best or most efficient 
> > > way to
> > implement it.
> > > We have fuego setup in a server and all our continuous integration 
> > > jobs (for multiple code bases) are setup in it. But the issue is 
> > > there are
> > too many builds happening, which puts pressure on the hard disk (so 
> > many read/writes) and also increases the docker container size (because of build logs).
> > >
> > > Hence I would like to know if we can distribute the load to 
> > > another server, for e.g. job will be created on our main server 
> > > but build will
> > happen on the agent on another server. The idea is to move out all 
> > cpu, memory, storage intensive activity to agents (other servers) on 
> > the network. The resultant logs can also be stored on those servers
> (outside docker container) so that docker size won't be growing rapidly.
> > >
> > > I tried creating a new node and setup an agent (connect via ssh) 
> > > but facing issues in installing the 'SSH Build Agents' plugin 
> > > because of
> > compatibility issues (jenkins version needs to be upgraded).
> > >
> > > If anyone already tried and has a solution for this issue please let me know.
> > >
> > > Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)?
> > >
> > > Regards,
> > > Dhinakar K.
> > > Samsung India, Bengaluru.
> > >
> > > On Thu, Feb 1, 2024 at 10:30 PM dhinakar k <dhinakar.k@gmail.com> wrote:
> > >>
> > >> Dear Fuego users,
> > >>
> > >> I have a requirement, so I thought of checking with the community 
> > >> if it was feasible and if so what is the best or most efficient 
> > >> way to
> > implement it.
> > >> We have fuego setup in a server and all our continuous 
> > >> integration jobs (for multiple code bases) are setup in it. But 
> > >> the issue is there are
> > too many builds happening, which puts pressure on the hard disk (so 
> > many read/writes) and also increases the docker container size (because of build logs).
> > >>
> > >> Hence I would like to know if we can distribute the load to 
> > >> another server, for e.g. job will be created on our main server 
> > >> but build will
> > happen on the agent on another server. The idea is to move out all 
> > cpu, memory, storage intensive activity to agents (other servers) on 
> > the network. The resultant logs can also be stored on those servers
> (outside docker container) so that docker size won't be growing rapidly.
> > >>
> > >> I tried creating a new node and setup an agent (connect via ssh) 
> > >> but facing issues in installing the 'SSH Build Agents' plugin 
> > >> because of
> > compatibility issues (jenkins version needs to be upgraded).
> > >>
> > >> If anyone already tried and has a solution for this issue please let me know.
> > >>
> > >> Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)?
> > >>
> > >> Regards,
> > >> Dhinakar K.
> > >> Samsung India, Bengaluru.
> 
> 
>