RE[7]: Distribute continuous integration (build) to servers outside docker container

["dhinakar.k" <dhinakar.k@samsung.com>]

Hi Tim,

Thanks for your prompt response.
I will explore more on this issue, otherwise without Jenkins security enabled, the fuego new version with Debian base distribution (bullseye) looks good.

Regards,
Dhinakar K.

-----Original Message-----
From: Bird, Tim [mailto:Tim.Bird@sony.com] 
Sent: Wednesday, May 8, 2024 5:26 AM
To: dhinakar.k <dhinakar.k@samsung.com>; fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org
Subject: RE: RE[6]: Distribute continuous integration (build) to servers outside docker container

Dhinakar,

I'm afraid you are doing things I have never done myself.  I'm not very familiar with Jenkins security.

Regarding the error you're seeing, however, I found this stackoverflow issue, which seems to have a few different ideas for dealing with a "crumb" error.

https://stackoverflow.com/questions/44711696/jenkins-403-no-valid-crumb-was-included-in-the-request

I will note that Fuego adds to the Jenkins url, so that the base url is https://.../fuego.  (that is, with an extra 'fuego' path element in the URL).  Maybe this is confusing the authentication code.

Sorry to not be more helpful.
 -- Tim

> -----Original Message-----
> From: dhinakar.k <dhinakar.k@samsung.com>
> Sent: Tuesday, May 7, 2024 6:15 AM
> To: Bird, Tim <Tim.Bird@sony.com>; fuego+owner@lists.linux.dev; 
> fuego@lists.linuxfoundation.org
> Subject: RE[6]: Distribute continuous integration (build) to servers 
> outside docker container
> 
> Hi Tim, Sorry for the delay in getting back to you. I set up fuego new 
> version with Debian base distribution (bullseye) and Jenkins (2. 414. 
> 1) support. Iam able to create 'nodes' with launch method as 'Launch 
> agents via SSH'. But agents won't ZjQcmQRYFpfptBannerStart Caution : This email originated from outside of Sony.
> Do not click links or open any attachments unless you recognize the sender and know the content is safe. Please report phishing if unsure.
> 
> ZjQcmQRYFpfptBannerEnd
> Hi Tim,
> 
> Sorry for the delay in getting back to you.
> I set up fuego new version with Debian base distribution (bullseye) and Jenkins (2.414.1) support.
> 
> Iam able to create 'nodes' with launch method as 'Launch agents via SSH'.
> But agents won't start, since the 'master' is looking into 
> '/var/lib/jenkins/.ssh/known_hosts' on the slave machine for key exchange, since default logged in user is 'jenkins'.
> 
> To experiment, I setup 'credentials' and enabled 'Dashboard -> Manage 
> Jenkins -> Security -> Authorization: Logged-in users can do anything', so that I could login with a pre-defined user which has access to both 'master' and 'slave' machines.
> But, I see below error as soon as I save the above settings, 'A 
> problem occurred while processing the request.
> Logging ID=8135f32e-e798-4153-b863-b438273750ba'
> 
> And, I see the below error if I try to login with previously configured user name and password.
> 
> HTTP ERROR 403 No valid crumb was included in the request
> URI:	/fuego/j_spring_security_check
> STATUS:	403
> MESSAGE:	No valid crumb was included in the request
> SERVLET:	Stapler
> 
> I am not sure what's happening here. It would help if you could provide some suggestion about how to proceed from here.
> Thanks in advance.
> 
> Regards,
> Dhinakar K.
> 
> -----Original Message-----
> From: dhinakar.k [mailto:dhinakar.k@samsung.com]
> Sent: Wednesday, March 27, 2024 4:09 PM
> To: 'Bird, Tim' <Tim.Bird@sony.com>; 'fuego+owner@lists.linux.dev' <fuego+owner@lists.linux.dev>; 'fuego@lists.linuxfoundation.org'
> <fuego@lists.linuxfoundation.org>
> Subject: RE[5]: Distribute continuous integration (build) to servers 
> outside docker container
> 
> Hi Tim,
> 
> 
> Thanks a lot for the release of fuego new version with Debian base distribution (bullseye) and Jenkins (2.414.1) support.
> 
> Regarding, distributing the load across server, I experimented and able to achieve it with the current fuego version itself.
> 
> While setting up the node I choose, 'Launch agent via execution of command on the controller' as the 'Launch method'.
> 
> And in the 'Launch command' I used, 'sshpass' command to connect to the Agent ('slave.jar') on a remote machine​​.
> 
> It is working fine, but still I would like to try the new version of fuego for user authentication, security features etc.
> 
> I will try it and let you know.
> 
> Thanks a lot once again.
> 
> 
> Note: I have already tested the 'Launch agents via SSH' on plain jenkins (recent version) as well and it is working fine.
> 
> Regards,
> Dhinakar K.
> 
> -----Original Message-----
> From: Bird, Tim [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, March 27, 2024 2:24 AM
> To: dhinakar.k <dhinakar.k@samsung.com>; fuego+owner@lists.linux.dev; 
> fuego@lists.linuxfoundation.org
> Subject: RE: RE[4]: Distribute continuous integration (build) to 
> servers outside docker container
> 
> Dhinakar,
> 
> I have committed some changes to the 'next' branch in the fuego and 
> fuego-core repositories for a more recent version of the Debian base 
> distribution (bullseye) and Jenkins (2.414.1)
> 
> I'm still testing and fixing issues, but many tests seem to work OK.
> 
> You can download this branch, using the following command:
>  $ git clone -b next 
> https://protect2.fireeye.com/v1/url?k=c205cdb3-a38ed885-c20446fc-74fe4
> 85cbff1- 
> 8f8985f51348034b&q=1&e=fd58435e-b89c-487a-8038-2c5394d50a38&u=https%3A
> %2F%2Fbitbucket.org%2Ffuegotest%2Ffuego.git
> 
> You can follow the instructions on the following pages, to perform installation of Fuego with this repository:
> 
> https://protect2.fireeye.com/v1/url?k=e17a6512-80f17024-e17bee5d-74fe4
> 85cbff1-44d3ec083952750c&q=1&e=fd58435e-b89c-487a-
> 8038-2c5394d50a38&u=https%3A%2F%2Ffuego.readthedocs.io%2Fen%2Flatest%2
> FQuickstart_Guide.html%23download-build-start-and-
> access
> https://protect2.fireeye.com/v1/url?k=14f10b08-757a1e3e-14f08047-74fe4
> 85cbff1-059e997d701ba98c&q=1&e=fd58435e-b89c-487a-
> 8038-2c5394d50a38&u=https%3A%2F%2Ffuego.readthedocs.io%2Fen%2Flatest%2
> FInstalling_Fuego.html%23downloading-a-different-
> branch
> 
> You will probably want to use a non-default docker image and container name.  See the following link for instructions:
> https://protect2.fireeye.com/v1/url?k=73a03639-122b230f-73a1bd76-74fe4
> 85cbff1-9f3005d17b0dba86&q=1&e=fd58435e-b89c-487a-
> 8038-2c5394d50a38&u=https%3A%2F%2Ffuego.readthedocs.io%2Fen%2Flatest%2
> FInstalling_Fuego.html%23using-an-different-container-
> name
> 
> In order to utilize distributed builds, I believe you will want agents 
> that direct jobs from a primary Jenkins instance, to parallel Jenkins instances running on other machines.
> I haven't don’t this myself, but I believe you will want to look at 
> how Jenkins uses ssh credentials, and uses "remoting" agents to execute things on other machines.
> 
> By default, Jenkins (as configured by Fuego), uses a node for each 
> Fuego board, with an agent (slave.jar) that is sent the job 
> information by the Jenkins host, and ends up calling the command for 
> the job definition (which calls 'ftc run-test ...').  This chain all 
> runs inside the docker container, on a single host machine.  However, I believe you can configure a node so that the job information is sent to a slave.jar running on a different machine.  This could either be to a machine running jenkins, or just running 'slave.jar' (or possibly 'agent.jar'), to execute the command for the job definition on that machine.
> 
> All this is just speculation, since I haven't done this myself.
> 
> Good luck.  Let me know how it goes.
>  -- Tim
> 
> > -----Original Message-----
> > From: dhinakar.k <dhinakar.k@samsung.com> Hi Tim,
> >
> > Thanks a lot for being kind enough to spend your precious time to analyze my storage issues.
> > My eyes were on '/var/lib/jenkins/workspace' when I copy pasted it to you.
> > I will look into it and resolve those issues. Thanks a lot once again.
> >
> > I will surely try the 'next' branch whenever it is available to check 'user authentication, security, and job distribution' features.
> > Thanks a have a nice weekend.
> >
> > Regards,
> > Dhinakar K.
> >
> > -----Original Message-----
> > From: Bird, Tim [mailto:Tim.Bird@sony.com]
> > Sent: Saturday, March 23, 2024 3:58 AM
> > To: dhinakar.k <dhinakar.k@samsung.com>; 
> > fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org
> > Subject: RE: RE[3]: Distribute continuous integration (build) to 
> > servers outside docker container
> >
> > Dhinakar,
> >
> > OK - here's a summary of your object counts and storage issues:
> > /var/lib/jenkins - 32G
> >   .../userContent - 1.8G
> >   .../jobs - 11G
> >   .../workspace - 19G
> >
> >  /fuego-rw - 13G
> >   ../buildzone - ~6G
> >   ../buildzone/X86-SSH-Test.default.Functional.kernel_build-x86-64 - 1.8G
> >   ../buildzone/Functional.LTP-x86_64 - 2.4G
> >
> > 6 nodes, 135 jobs, 78 runs
> >
> > docker image: 2.41G?
> > (or is it 173 GB?)
> >
> > This is confusing.  Your latest fuego docker container is 2.41G, but 
> > you have /var/lib/jenkins that is about 32G (and that should all be in your container image).
> >
> > It looks like the builds for the kernel and for LTP are taking the most space.
> >
> > I have no idea why your /var/lib/jenkins/jobs directory would be so big (11G) Maybe do:
> > $ du -sh /var/lib/jenkins/jobs/* | sort -sh and see which jobs have a lot of info.
> >
> > Also, I don't know what's going on with your /var/lib/jenkins/workspace (19G) I don't even have that directory on my system.
> >
> > My stats look quite different:
> > I have 13 nodes, 387 jobs and 1129 runs in my most active Fuego container, my /var/lib/Jenkins is only 204M.
> >
> > My /fuego-rw/buildzone is 64G, due to lots of instances of LTP (and something very funky about one Interbench build):
> > 1.2G	ren1.latest-smoke.Functional.LTP-poky-aarch64
> > 1.7G	Functional.kselftest-x86_64
> > 1.7G	rpi3-2.default.Functional.kernel_build-aarch64
> > 1.9G	Functional.LTP_one_at_a_time-debian-armhf
> > 2.0G	Functional.raspi_kinstall-debian-armhf
> > 2.3G	bbb.latest-smoke.Functional.LTP-debian-armhf
> > 2.4G	Functional.LTP-x86_64
> > 2.4G	Functional.LTP_one_at_a_time-x86_64
> > 2.5G	Functional.LTP-poky-aarch64
> > 2.5G	Functional.LTP_one_at_a_time-aarch64
> > 2.5G	Functional.LTP_one_at_a_time-poky-aarch64
> > 3.0G	min1.latest-smoke.Functional.LTP-x86_64
> > 3.0G	min1.latest.Functional.LTP-x86_64
> > 3.2G	rpi3-2.latest-smoke.Functional.LTP-aarch64
> > 32G	Benchmark.Interbench-x86_64
> >
> > There might be stuff in /var/lib/jenkins/workspace you could get rid of.
> >
> > I have a lot more runs on my system, but very many of those have 
> > very short run.json files, as they are for tests with small results. 
> > I believe the largest run.json and consolelog.txt files are for LTP.  
> > If you have a lot of those, maybe you can archive old ones to reduce 
> > disk
> usage and improve performance.
> >
> > In any event, is there any particular operation which is taking a 
> > long time to complete, due to the docker size (or fuego directory) being too big or taking too much space?
> >
> > Finally, I'm still working on the new docker container (with more 
> > recent versions of Debian and Jenkins).  I've already fixed a few 
> > bugs that I found, but there are a lot of old Fuego packages that 
> > don't compile with newer toolchains.  There are workarounds for 
> > this, but it will
> take a while to implement.
> >
> > My plan is to publish my working branch for this work on bitbucket, as a 'next' branch, even though not all problems are resolved.
> > You could then build a container using that (if you want to), and 
> > check if the new version of Jenkins has the features you need for user authentication, security, and job distribution.
> >
> > I can give you some install steps for using a 'next' branch, if you are interested in trying that.
> >
> > Sorry this has taken so long.
> >  -- Tim
> >
> >
> >
> > > -----Original Message-----
> > > From: dhinakar.k <dhinakar.k@samsung.com> Hi Tim,
> > >
> > > Thanks for your prompt response.
> > > Please find the details you requested below.
> > >
> > > # du -sh /fuego-rw/logs/* | sort -h
> > > 4.0K    /fuego-rw/logs/README
> > > 8.0K    /fuego-rw/logs/logruns
> > > 56K     /fuego-rw/logs/Benchmark.gtkperf
> > > 60K     /fuego-rw/logs/Functional.glib
> > > 96K     /fuego-rw/logs/Benchmark.GLMark
> > > 228K    /fuego-rw/logs/theme
> > > 284K    /fuego-rw/logs/Benchmark.dbench4
> > > 284K    /fuego-rw/logs/Benchmark.himeno
> > > 284K    /fuego-rw/logs/Benchmark.iperf
> > > 284K    /fuego-rw/logs/Benchmark.x11perf
> > > 288K    /fuego-rw/logs/Benchmark.linpack
> > > 288K    /fuego-rw/logs/Functional.aiostress
> > > 288K    /fuego-rw/logs/Functional.bc
> > > 288K    /fuego-rw/logs/Functional.crashme
> > > 288K    /fuego-rw/logs/Functional.fontconfig
> > > 288K    /fuego-rw/logs/Functional.hello_world
> > > 288K    /fuego-rw/logs/Functional.stress
> > > 288K    /fuego-rw/logs/Functional.synctest
> > > 288K    /fuego-rw/logs/Functional.zlib
> > > 292K    /fuego-rw/logs/Benchmark.Stream
> > > 292K    /fuego-rw/logs/Benchmark.netperf
> > > 292K    /fuego-rw/logs/Benchmark.tiobench
> > > 296K    /fuego-rw/logs/Functional.ipv6connect
> > > 304K    /fuego-rw/logs/Benchmark.nbench_byte
> > > 304K    /fuego-rw/logs/Benchmark.netpipe
> > > 312K    /fuego-rw/logs/Benchmark.fio
> > > 312K    /fuego-rw/logs/Functional.scrashme
> > > 316K    /fuego-rw/logs/Benchmark.Java
> > > 344K    /fuego-rw/logs/Functional.bzip2
> > > 352K    /fuego-rw/logs/Benchmark.IOzone
> > > 352K    /fuego-rw/logs/Functional.jpeg
> > > 356K    /fuego-rw/logs/Benchmark.OpenSSL
> > > 372K    /fuego-rw/logs/Functional.netperf
> > > 396K    /fuego-rw/logs/Functional.expat
> > > 400K    /fuego-rw/logs/Benchmark.lmbench2
> > > 404K    /fuego-rw/logs/Functional.kernel_build
> > > 424K    /fuego-rw/logs/Functional.rmaptest
> > > 524K    /fuego-rw/logs/image
> > > 536K    /fuego-rw/logs/Benchmark.Interbench
> > > 552K    /fuego-rw/logs/Benchmark.ebizzy
> > > 552K    /fuego-rw/logs/Benchmark.signaltest
> > > 556K    /fuego-rw/logs/Functional.pi_tests
> > > 588K    /fuego-rw/logs/Benchmark.Whetstone
> > > 740K    /fuego-rw/logs/Functional.ft2demos
> > > 808K    /fuego-rw/logs/Benchmark.Dhrystone
> > > 1.1M    /fuego-rw/logs/Benchmark.cyclictest
> > > 1.1M    /fuego-rw/logs/Benchmark.ffsb
> > > 1.2M    /fuego-rw/logs/Benchmark.bonnie
> > > 1.6M    /fuego-rw/logs/Benchmark.hackbench
> > > 2.1M    /fuego-rw/logs/Functional.OpenSSL
> > > 7.5M    /fuego-rw/logs/Extract
> > > 11M     /fuego-rw/logs/Functional.linus_stress
> > > 12M     /fuego-rw/logs/Functional.LTP
> > >
> > > # ftc list-nodes | wc -l
> > > 6
> > >
> > > Jenkins nodes in this system:
> > >    ARM64-SSH-Test
> > >    ARM64-Serial-Test
> > >    SAPIENT-MAIN-Server-Node
> > >    X86-SSH-Test
> > >    X86-Serial-Test
> > >
> > > # ftc list-jobs | wc -l
> > > 135
> > >
> > > # ftc list-runs | wc -l
> > > 78
> > >
> > > Regards,
> > > Dhinakar K.
> > >
> > > -----Original Message-----
> > > From: Bird, Tim [mailto:Tim.Bird@sony.com]
> > > Sent: Saturday, March 16, 2024 12:12 AM
> > > To: dhinakar.k <dhinakar.k@samsung.com>;
> > > fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org
> > > Subject: RE: RE[3]: Distribute continuous integration (build) to 
> > > servers outside docker container
> > >
> > > Dhinkar,
> > >
> > > Can you send me the info for these as well:
> > >
> > > # du -sh /fuego-rw/logs/* | sort -h
> > >
> > >
> > > # ftc list-nodes | wc -l
> > >
> > >
> > > # ftc list-jobs | wc -l
> > >
> > >
> > > # ftc list-runs | wc -l
> > >
> > >
> > >  -- Tim
> > >
> > > > -----Original Message-----
> > > > From: dhinakar.k <dhinakar.k@samsung.com>
> > > > Sent: Friday, March 15, 2024 9:38 AM
> > > > To: Bird, Tim <Tim.Bird@sony.com>; fuego+owner@lists.linux.dev; 
> > > > fuego@lists.linuxfoundation.org
> > > > Subject: RE[3]: Distribute continuous integration (build) to 
> > > > servers outside docker container
> > > >
> > > > Hi Tim, Thanks a lot for your response. Please find below the 
> > > > details that you requested. # du -sh /var/lib/Jenkins 32G 
> > > > /var/lib/Jenkins # du -sh
> > > > /var/lib/jenkins/* | sort -h 0 /var/lib/jenkins/secret. key.
> > > > not-so-secret 4. 0K /var/lib/jenkins/com. cloudbees. hudson. plugins.
> > > > folder. config. AbstractFolderConfiguration. xml 
> > > > ZjQcmQRYFpfptBannerStart Caution : This email originated from outside of Sony.
> > > > Do not click links or open any attachments unless you recognize 
> > > > the sender and know the content is safe. Please report phishing 
> > > > if
> unsure.
> > > >
> > > > ZjQcmQRYFpfptBannerEnd
> > > > Hi Tim,
> > > >
> > > > Thanks a lot for your response.
> > > > Please find below the details that you requested.
> > > >
> > > > # du -sh /var/lib/Jenkins
> > > > 32G     /var/lib/Jenkins
> > > >
> > > > # du -sh /var/lib/jenkins/* | sort -h
> > > > 0       /var/lib/jenkins/secret.key.not-so-secret
> > > > 4.0K    /var/lib/jenkins/com.cloudbees.hudson.plugins.folder.config.AbstractFolderConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/com.mtvi.plateng.hudson.ldap.LdapMailAddressResolver.xml
> > > > 4.0K    /var/lib/jenkins/com.orctom.jenkins.plugin.buildtimestamp.BuildTimestampWrapper.xml
> > > > 4.0K    /var/lib/jenkins/credentials.xml
> > > > 4.0K    /var/lib/jenkins/hudson.maven.MavenModuleSet.xml
> > > > 4.0K    /var/lib/jenkins/hudson.model.UpdateCenter.xml
> > > > 4.0K    /var/lib/jenkins/hudson.plugins.emailext.ExtendedEmailPublisher.xml
> > > > 4.0K    /var/lib/jenkins/hudson.plugins.git.GitSCM.xml
> > > > 4.0K    /var/lib/jenkins/hudson.plugins.git.GitTool.xml
> > > > 4.0K    /var/lib/jenkins/hudson.plugins.sidebar_link.SidebarLinkPlugin.xml
> > > > 4.0K    /var/lib/jenkins/hudson.plugins.sonar.SonarGlobalConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/hudson.plugins.timestamper.TimestamperConfig.xml
> > > > 4.0K    /var/lib/jenkins/hudson.tasks.Mailer.xml
> > > > 4.0K    /var/lib/jenkins/hudson.tasks.Shell.xml
> > > > 4.0K    /var/lib/jenkins/hudson.triggers.SCMTrigger.xml
> > > > 4.0K    /var/lib/jenkins/identity.key.enc
> > > > 4.0K    /var/lib/jenkins/io.jenkins.plugins.junit.storage.JunitTestResultStorageConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/jenkins.fingerprints.GlobalFingerprintConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/jenkins.install.InstallUtil.lastExecVersion
> > > > 4.0K    /var/lib/jenkins/jenkins.install.UpgradeWizard.state
> > > > 4.0K    /var/lib/jenkins/jenkins.model.ArtifactManagerConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/jenkins.model.GlobalBuildDiscarderConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/jenkins.plugins.msginject.MsgInjectConfig.xml
> > > > 4.0K    /var/lib/jenkins/jenkins.security.ResourceDomainConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/jenkins.security.apitoken.ApiTokenPropertyConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/jenkins.tasks.filters.EnvVarsFilterGlobalConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/jenkins.telemetry.Correlator.xml
> > > > 4.0K    /var/lib/jenkins/net.plavcak.jenkins.plugins.scmskip.SCMSkipBuildWrapper.xml
> > > > 4.0K    /var/lib/jenkins/net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.xml
> > > > 4.0K    /var/lib/jenkins/nodeMonitors.xml
> > > > 4.0K    /var/lib/jenkins/org.codefirst.SimpleThemeDecorator.xml
> > > > 4.0K    /var/lib/jenkins/org.jenkinsCi.plugins.projectDescriptionSetter.DescriptionSetterWrapper.xml
> > > > 4.0K    /var/lib/jenkins/org.jenkinsci.plugins.ansible_tower.AnsibleTower.xml
> > > > 4.0K    /var/lib/jenkins/org.jenkinsci.plugins.corsfilter.AccessControlsFilter.xml
> > > > 4.0K    /var/lib/jenkins/org.jenkinsci.plugins.lucene.search.config.SearchBackendConfiguration.xml
> > > > 4.0K    /var/lib/jenkins/org.jenkinsci.plugins.workflow.flow.GlobalDefaultFlowDurabilityLevel.xml
> > > > 4.0K    /var/lib/jenkins/org.quality.gates.jenkins.plugin.GlobalConfig.xml
> > > > 4.0K    /var/lib/jenkins/queue.xml
> > > > 4.0K    /var/lib/jenkins/queue.xml.bak
> > > > 4.0K    /var/lib/jenkins/scriptApproval.xml
> > > > 4.0K    /var/lib/jenkins/secret.key
> > > > 4.0K    /var/lib/jenkins/sidebar-link.xml
> > > > 12K     /var/lib/jenkins/gerrit-trigger.xml
> > > > 16K     /var/lib/jenkins/hudson.plugins.ansicolor.AnsiColorBuildWrapper.xml
> > > > 20K     /var/lib/jenkins/config.xml
> > > > 44K     /var/lib/jenkins/nodes
> > > > 72K     /var/lib/jenkins/secrets
> > > > 356K    /var/lib/jenkins/users
> > > > 1.7M    /var/lib/jenkins/logs
> > > > 1.8M    /var/lib/jenkins/luceneIndex
> > > > 3.6M    /var/lib/jenkins/updates
> > > > 3.9M    /var/lib/jenkins/plugins_backup
> > > > 316M    /var/lib/jenkins/plugins
> > > > 316M    /var/lib/jenkins/plugins_not_working
> > > > 372M    /var/lib/jenkins/plugins_old_2ndFeb2024
> > > > 376M    /var/lib/jenkins/plugins_working_2ndFeb2024
> > > > 1.8G    /var/lib/jenkins/userContent
> > > > 11G     /var/lib/jenkins/jobs
> > > > 19G     /var/lib/jenkins/workspace
> > > >
> > > > # du -sh /fuego-rw
> > > > 13G     /fuego-rw
> > > >
> > > > # du -sh /fuego-rw/buildzone/* | sort -h # du -sh 
> > > > /fuego-rw/logs/*
> > > > | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc
> > > > list-runs | wc -
> > l
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Dhrystone-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-aarch64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.IOzone-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Interbench-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Java-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.OpenSSL-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Stream-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Whetstone-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.bonnie-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.cyclictest-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.dbench4-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ebizzy-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ffsb-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.fio-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.gtkperf-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.hackbench-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.himeno-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.iperf-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.linpack-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.lmbench2-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.nbench_byte-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netperf-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netpipe-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.signaltest-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.tiobench-x86_64
> > > > 0       /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.x11perf-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Dhrystone-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.GLMark-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.IOzone-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Interbench-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Java-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.OpenSSL-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Stream-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Whetstone-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.bonnie-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.cyclictest-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.dbench4-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ebizzy-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ffsb-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.fio-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.gtkperf-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.hackbench-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.himeno-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.iperf-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.linpack-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.lmbench2-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.nbench_byte-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netperf-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netpipe-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.signaltest-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.tiobench-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.x11perf-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.LTP-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.OpenSSL-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.aiostress-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bc-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bzip2-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.crashme-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.expat-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.fontconfig-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ft2demos-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.glib-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.hello_world-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ipv6connect-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.jpeg-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.linus_stress-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.netperf-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.pi_tests-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.rmaptest-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.scrashme-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.stress-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.synctest-x86_64
> > > > 0       /fuego-rw/buildzone/X86-SSH-Test.default.Functional.zlib-x86_64
> > > > 4.0K    /fuego-rw/buildzone/README
> > > > 12K     /fuego-rw/buildzone/Functional.bc-x86_64
> > > > 32K     /fuego-rw/buildzone/Functional.hello_world-x86_64
> > > > 32K     /fuego-rw/buildzone/Functional.linus_stress-x86_64
> > > > 32K     /fuego-rw/buildzone/Functional.synctest-x86_64
> > > > 36K     /fuego-rw/buildzone/Functional.rmaptest-x86_64
> > > > 48K     /fuego-rw/buildzone/Benchmark.himeno-x86_64
> > > > 48K     /fuego-rw/buildzone/Benchmark.linpack-x86_64
> > > > 48K     /fuego-rw/buildzone/Functional.ipv6connect-x86_64
> > > > 60K     /fuego-rw/buildzone/Benchmark.Stream-x86_64
> > > > 76K     /fuego-rw/buildzone/Benchmark.Whetstone-x86_64
> > > > 76K     /fuego-rw/buildzone/Benchmark.ebizzy-x86_64
> > > > 84K     /fuego-rw/buildzone/Functional.aiostress-x86_64
> > > > 168K    /fuego-rw/buildzone/Benchmark.Dhrystone-x86_64
> > > > 240K    /fuego-rw/buildzone/Benchmark.tiobench-x86_64
> > > > 244K    /fuego-rw/buildzone/Functional.crashme-x86_64
> > > > 488K    /fuego-rw/buildzone/Benchmark.Interbench-x86_64
> > > > 576K    /fuego-rw/buildzone/Functional.scrashme-x86_64
> > > > 760K    /fuego-rw/buildzone/Benchmark.hackbench-x86_64
> > > > 816K    /fuego-rw/buildzone/Benchmark.signaltest-x86_64
> > > > 868K    /fuego-rw/buildzone/Functional.pi_tests-x86_64
> > > > 900K    /fuego-rw/buildzone/Benchmark.netpipe-x86_64
> > > > 904K    /fuego-rw/buildzone/Benchmark.cyclictest-x86_64
> > > > 1004K   /fuego-rw/buildzone/Functional.stress-x86_64
> > > > 1.3M    /fuego-rw/buildzone/Benchmark.ffsb-x86_64
> > > > 1.6M    /fuego-rw/buildzone/Benchmark.GLMark-aarch64
> > > > 1.6M    /fuego-rw/buildzone/Benchmark.GLMark-x86_64
> > > > 1.7M    /fuego-rw/buildzone/Benchmark.nbench_byte-x86_64
> > > > 2.1M    /fuego-rw/buildzone/Benchmark.iperf-x86_64
> > > > 2.3M    /fuego-rw/buildzone/Benchmark.gtkperf-x86_64
> > > > 2.4M    /fuego-rw/buildzone/Benchmark.bonnie-x86_64
> > > > 2.5M    /fuego-rw/buildzone/Benchmark.x11perf-x86_64
> > > > 2.9M    /fuego-rw/buildzone/Functional.bzip2-x86_64
> > > > 3.0M    /fuego-rw/buildzone/Functional.zlib-x86_64
> > > > 3.1M    /fuego-rw/buildzone/Benchmark.IOzone-x86_64
> > > > 3.5M    /fuego-rw/buildzone/Functional.jpeg-x86_64
> > > > 5.2M    /fuego-rw/buildzone/Benchmark.lmbench2-x86_64
> > > > 6.1M    /fuego-rw/buildzone/Benchmark.netperf-x86_64
> > > > 6.1M    /fuego-rw/buildzone/Functional.netperf-x86_64
> > > > 7.0M    /fuego-rw/buildzone/Functional.fontconfig-x86_64
> > > > 9.1M    /fuego-rw/buildzone/Benchmark.fio-x86_64
> > > > 29M     /fuego-rw/buildzone/Functional.expat-x86_64
> > > > 33M     /fuego-rw/buildzone/Benchmark.dbench4-x86_64
> > > > 34M     /fuego-rw/buildzone/Functional.ft2demos-x86_64
> > > > 49M     /fuego-rw/buildzone/Functional.glib-x86_64
> > > > 74M     /fuego-rw/buildzone/Benchmark.OpenSSL-x86_64
> > > > 74M     /fuego-rw/buildzone/Functional.OpenSSL-x86_64
> > > > 160M    /fuego-rw/buildzone/Benchmark.Java-x86_64
> > > > 1.8G    /fuego-rw/buildzone/X86-SSH-Test.default.Functional.kernel_build-x86_64
> > > > 2.4G    /fuego-rw/buildzone/Functional.LTP-x86_64
> > > >
> > > > Outside the container, can you run:
> > > > $ docker image ls
> > > > nxo-container-backup-2ndsep2021          latest         b6a3d16508e8   2 years ago     173GB
> > > > fuego-base-image-20thaug2019-container   latest         b2cf22891ac6   4 years ago     2.41GB
> > > > fuegocontainerbackup-4thmar2019          latest         04112c383476   5 years ago     3.34GB
> > > > fuego-piper-image-22ndnov2021           latest         0121b1c342e6   6 years ago     10.7GB
> > > > fuego-xyz-image-22ndnov2021              latest         0bf6b942db70   6 years ago     5.8GB
> > > > fuego-dart-image-22ndnov2021           latest         1d2c6a99dc0a   6 years ago     1.66GB
> > > > fuego-pavv-image-22ndnov2021             latest         ba71ede1e872   6 years ago     4.63GB
> > > > fuego-gpu-image-22ndnov2021           latest         42718d86dd09   6 years ago     3.41GB
> > > >
> > > > Regarding below request, I just want to lock the Jenkins user 
> > > > interface/ dashboard, so that none can see all the jobs, 
> > > > results, logs and other details without logging in. So to start 
> > > > with even one user with
> > > login credentials will help.
> > > > But I don't know how Jenkins credentials relate to user accounts.
> > > > I haven't ever set up Jenkins to handle multiple user accounts.
> > > > Indeed, in my current 2.414.1 Jenkins setup I can see an option 
> > > > to list "People" (which I presume is registered user accounts), 
> > > > but I can find no way
> > > to create or manage new user accounts.
> > > >
> > > > Thanks a lot for all other details, they were very helpful.
> > > >
> > > > Regards,
> > > > Dhinakar K.
> > > >
> > > > -----Original Message-----
> > > > From: Bird, Tim [mailto:Tim.Bird@sony.com]
> > > > Sent: Wednesday, March 13, 2024 5:47 AM
> > > > To: dhinakar.k <dhinakar.k@samsung.com>;
> > > > fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org
> > > > Subject: RE: HTML message rejected: Re: Distribute continuous 
> > > > integration (build) to servers outside docker container
> > > >
> > > > Hey Dhinakar,
> > > >
> > > > Sorry to not respond sooner.  There was a lot to process in your 
> > > > email, and some of it I don't have answers for, but I'll try to give some status and feedback where I can.
> > > >
> > > > > -----Original Message-----
> > > > > From: dhinakar.k <dhinakar.k@samsung.com> Hi Tim,
> > > > >
> > > > > I had sent a reply to your response on 3rd Feb.2024, not sure if it had reached you.
> > > > > I am posting the same message below again.
> > > > >
> > > > > Thanks a lot for your prompt reply.
> > > > > Iam looking forward to try the updated docker container with 
> > > > > an upgraded distribution base from Debian stretch to Debian 
> > > > > bullseye, and from a Jenkins version of 2.249.3 to 2.414.1.
> > > > > Please let me
> > > > whenever it is available.
> > > >
> > > > OK - Will do.  I haven't been able to allocate time for testing 
> > > > and fixing it, so I didn't want to push it to the master branch.
> > > > If I pushed something to a testing branch, would you be able to 
> > > > download from
> > > there and test it?
> > > >
> > > > One issue that came up is that some of the toolchains for some 
> > > > of my old boards/distros are not supported (ie don't work) in Debian bullseye, and so I was looking for solutions for that.
> > > >
> > > > > We are migrating to another server, so Iam looking for best 
> > > > > options to transfer all the content to the new server. Since 
> > > > > the docker container size is huge I want to trim it down by removing unwanted old logs.
> > > >
> > > > It would be good to know where the container size is coming from.
> > > > Is it in the Fuego build directories, the Fuego run directories 
> > > > (where Fuego logs and run artifacts are stored), or in the 
> > > > Jenkins build directories (where Jenkins logs and artifacts 
> > > > are)?  The materials under /fuego-rw should be on the host 
> > > > system, and not inside the container
> > > itself (since these are volume mounts).  But I don't know how this 
> > > space is accounted to the container (if any).  It shouldn't be in any container image.
> > > >
> > > > The materials under /var/lib/jenkins are inside the container image.
> > > >
> > > > Can you run the following commands, inside your container, and send the results:
> > > >
> > > > # du -sh /var/lib/jenkins
> > > > # du -sh /var/lib/jenkins/* | sort -h # du -sh /fuego-rw # du 
> > > > -sh
> > > > /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | sort 
> > > > -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc 
> > > > list-runs | wc -l
> > > >
> > > > Outside the container, can you run:
> > > > $ docker image ls
> > > > (and remove any lines not related to fuego images)
> > > >
> > > > But I don't know how Jenkins credentials relate to user accounts.
> > > > I haven't ever set up Jenkins to handle multiple user accounts.
> > > > Indeed, in my current 2.414.1 Jenkins setup I can see an option 
> > > > to list "People" (which I presume is registered user accounts), 
> > > > but I can find no way
> > > to create or manage new user accounts.
> > > >
> > > > > Also, I am worried about server crash and hard disk crash due 
> > > > > to so many read/write cycles (that happens during continuous
> integration).
> > > > > Hence looking for the best way to back up the container and 
> > > > > also to distribute the load & read/write cycles to another 
> > > > > server (via an ssh agent). I tried with plain Jenkins via ssh 
> > > > > credentials plugin, it seems feasible. Hence if fuego upgrades 
> > > > > to debian bullseye it would be really
> > > > helpful.
> > > > OK - I'll keep plugging away at this.
> > > >
> > > > >
> > > > > Also, would like to know, when login credentials support will 
> > > > > be available in fuego? Because as a security measure we are 
> > > > > supposed to have login credentials for each user. Will be 
> > > > > available after debian
> > > > upgrade?
> > > >
> > > > I believe you are referring to Jenkins credentials?  It appears 
> > > > that Jenkins supports credentials in version 2.414, which is the 
> > > > new version of Jenkins in the new docker container (with the 
> > > > Debian upgrade).  So
> > > I believe the answer is yes.
> > > >
> > > > But I don't know how Jenkins credentials relate to user accounts.
> > > > I haven't ever set up Jenkins to handle multiple user accounts.
> > > > Indeed, in my current 2.414.1 Jenkins setup I can see an option 
> > > > to list "People" (which I presume is registered user accounts), 
> > > > but I can find no way
> > > to create or manage new user accounts.
> > > >
> > > > > With respect to removing the logs I will figure out the best 
> > > > > way as I see so many options but not sure if that will reduce 
> > > > > the container size directly or need to run some container 
> > > > > commands to trim it down. I
> > > > will explore that as well and email my observations in this forum.
> > > >
> > > > I have considered adding an "rm-run" sub-command to ftc, which would allow you to remove old runs.
> > > > However, I'd like to see if it's the run data that is causing the size problem that you are seeing.
> > > >
> > > > If you want to archive the data for an individual run, then you 
> > > > can use 'ftc package-run' and move the resulting package to some 
> > > > long-term storage device (before, ostensibly, removing that run 
> > > > from the
> > > > system.)
> > > >
> > > > I hope this is helpful.
> > > >  -- Tim
> > > >
> > > >
> > > >
> > > > >
> > > > > Regards,
> > > > > Dhinakar K.
> > > > >
> > > > > -----Original Message-----
> > > > > From: Bird, Tim [mailto:Tim.Bird@sony.com]
> > > > > Sent: Saturday, February 3, 2024 3:44 AM
> > > > > To: dhinakar k <dhinakar.k@gmail.com>;
> > > > > fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org
> > > > > Subject: RE: HTML message rejected: Re: Distribute continuous 
> > > > > integration (build) to servers outside docker container
> > > > >
> > > > > Dhinakar,  See my answers inline below..
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: dhinakar k <dhinakar.k@gmail.com> Dear Fuego users,
> > > > > >
> > > > > > I have a requirement, so I thought of checking with the 
> > > > > > community if it was feasible and if so what is the best or 
> > > > > > most efficient way to implement it.
> > > > > > We have fuego setup in a server and all our continuous 
> > > > > > integration jobs (for multiple code bases) are setup in it.
> > > > > > But the issue is there are too many builds happening, which 
> > > > > > puts pressure on the hard disk (so many read/writes) and 
> > > > > > also increases the docker container size (because of build logs).
> > > > > >
> > > > > > Hence I would like to know if we can distribute the load to 
> > > > > > another server, for e.g. job will be created on our main 
> > > > > > server but build will happen on the agent on another server.
> > > > > > The idea is to move out all cpu, memory, storage intensive 
> > > > > > activity to agents (other
> > > > > > servers) on the network. The resultant logs can also be 
> > > > > > stored on those servers (outside docker container) so that 
> > > > > > docker size won't be growing rapidly.
> > > > > >
> > > > > > I tried creating a new node and setup an agent (connect via
> > > > > > ssh) but facing issues in installing the 'SSH Build Agents'
> > > > > > plugin because of compatibility issues (jenkins version needs to be upgraded).
> > > > > >
> > > > > > If anyone already tried and has a solution for this issue please let me know.
> > > > >
> > > > > Unfortunately, I have not tried this, so I'm not familiar with how Jenkins supports distributed jobs.
> > > > >
> > > > > But I have a few ideas to share.
> > > > >
> > > > > First, I'm working now on an update to the docker container, 
> > > > > from a distribution base of Debian stretch to Debian bullseye, 
> > > > > and from a Jenkins version of 2.249.3 to 2.414.1 (thanks to a 
> > > > > patch provided by Fuego user Yoichi Tachibana).  I have 
> > > > > applied the patches, but found
> > > > some issues with the build instructions for some packages 
> > > > working in Debian bullseye.  So I have not committed the changes 
> > > > to
> > > > > the Fuego master branch yet.   I hope to be able to do this soon.
> > > > >
> > > > > This might help with the issue of not being able to use the ssh agent plugin for Jenkins.
> > > > >
> > > > > Second, it may be possible to do some kind of proxy job 
> > > > > submission thing, by creating proxy jobs on the main Jenkins 
> > > > > server, but having the jobs actually executed ("built", in 
> > > > > Jenkins
> > > > > terminology) on secondary
> > > > servers.
> > > > >
> > > > > There are a number of ways of supporting remote execution, and 
> > > > > how you implement it will determine where the CPU cycles and 
> > > > > memory usage is during the job, and where the logs end up 
> > > > > living (and taking up
> > > > > space.)
> > > > >
> > > > > If you are using 'ttc' as your transport layer, recent 
> > > > > versions of ttc have support for remote boards via ssh (configured in ttc.conf).  But that would leave logs on the master server.
> > > > >
> > > > > There are multiple ways to configure Fuego on the secondary 
> > > > > servers
> > > > > - for example using Docker containers or installing Fuego natively.
> > > > > In any event, it may be possible to set up proxy jobs on the 
> > > > > main server that
> > > > call 'ftc' on the secondary
> > > > > servers, to run the jobs of interest.   Depending on how this was structured
> > > > > you may end up with build artifacts and logs on the secondary 
> > > > > system, and only Jenkins artifacts and logs on the main system (which should save space).
> > > > >
> > > > > It might be good to get a more detailed description of your Jenkins setup to be able to understand the options available.
> > > > >
> > > > > >
> > > > > > Also, what is the best way to backup the container after 
> > > > > > trimming it down (remove unwanted files/logs etc. and reduce it's size)?
> > > > >
> > > > > I'm not sure how to backup a container in Docker.  There's the 'docker container commit'
> > > > > command, but I haven't used that.
> > > > >
> > > > > In general, to trim down you data usage you will want to perform a few operations.
> > > > > Inside Jenkins, you can delete individual builds.  You can 
> > > > > likely automate this (not have to do it manually  through the 
> > > > > UI), by using the remote API.  That should work for the 
> > > > > material that is inside the container.  Lots of data, however, 
> > > > > including the build artifacts and the
> > > > logs are natively on the host, and are only present in the container through bind mounts.
> > > > >
> > > > > This means you can remove this data directly using file operations (e.g. rm) on the host.
> > > > > There are a few nuances to this, however, as some of the build directories are referenced via symlinks.
> > > > >
> > > > > 'ftc' currently supports removing jobs and nodes.  I don't 
> > > > > recall off the top of my head if this includes removing all 
> > > > > the builds for a job or
> > > not.
> > > > > But in any case, it sounds like you only want to remove some of the builds/runs, not the jobs themselves.
> > > > > We could possibly support 'ftc rm-run', which could remove specific runs.  'ftc query-runs'
> > > > > could be used to generate a list of runs (e.g. that were created before a certain date).
> > > > >
> > > > > Please provide a few more details, and maybe we can talk through and/or develop some solutions that will help meet your needs.
> > > > >
> > > > >  -- Tim
> > > > >
> > > > > >
> > > > > > On Thu, Feb 1, 2024 at 11:52 PM <fuego+owner@lists.linux.dev> wrote:
> > > > > > >
> > > > > > > Greetings!
> > > > > > >
> > > > > > > This is the mlmmj program managing the 
> > > > > > > <fuego@lists.linux.dev> mailing list.
> > > > > > >
> > > > > > > Your message to <fuego@lists.linux.dev> was not delivered 
> > > > > > > to the list because it contained a HTML part. Only 
> > > > > > > text/plain messages are allowed on this list.
> > > > > > >
> > > > > > > Please configure your mail client to only send plain text mail.
> > > > > > >
> > > > > > > For your reference, the rejected message follows below.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ---------- Forwarded message ----------
> > > > > > > From: dhinakar k <dhinakar.k@gmail.com>
> > > > > > > To: fuego+help@lists.linux.dev, 
> > > > > > > fuego@lists.linuxfoundation.org, "Bird, Timothy"
> > > > > > > <Tim.Bird@sony.com>
> > > > > > > Cc:
> > > > > > > Bcc:
> > > > > > > Date: Thu, 1 Feb 2024 23:51:30 +0530
> > > > > > > Subject: Re: Distribute continuous integration (build) to 
> > > > > > > servers outside docker container Dear Fuego users,
> > > > > > >
> > > > > > > I have a requirement, so I thought of checking with the 
> > > > > > > community if it was feasible and if so what is the best or 
> > > > > > > most efficient way to
> > > > > > implement it.
> > > > > > > We have fuego setup in a server and all our continuous 
> > > > > > > integration jobs (for multiple code bases) are setup in it.
> > > > > > > But the issue is there are
> > > > > > too many builds happening, which puts pressure on the hard 
> > > > > > disk (so many read/writes) and also increases the docker container size (because of build logs).
> > > > > > >
> > > > > > > Hence I would like to know if we can distribute the load 
> > > > > > > to another server, for e.g. job will be created on our 
> > > > > > > main server but build will
> > > > > > happen on the agent on another server. The idea is to move 
> > > > > > out all cpu, memory, storage intensive activity to agents 
> > > > > > (other
> > > > > > servers) on the network. The resultant logs can also be 
> > > > > > stored on those servers
> > > > > (outside docker container) so that docker size won't be growing rapidly.
> > > > > > >
> > > > > > > I tried creating a new node and setup an agent (connect 
> > > > > > > via
> > > > > > > ssh) but facing issues in installing the 'SSH Build Agents'
> > > > > > > plugin because of
> > > > > > compatibility issues (jenkins version needs to be upgraded).
> > > > > > >
> > > > > > > If anyone already tried and has a solution for this issue please let me know.
> > > > > > >
> > > > > > > Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)?
> > > > > > >
> > > > > > > Regards,
> > > > > > > Dhinakar K.
> > > > > > > Samsung India, Bengaluru.
> > > > > > >
> > > > > > > On Thu, Feb 1, 2024 at 10:30 PM dhinakar k <dhinakar.k@gmail.com> wrote:
> > > > > > >>
> > > > > > >> Dear Fuego users,
> > > > > > >>
> > > > > > >> I have a requirement, so I thought of checking with the 
> > > > > > >> community if it was feasible and if so what is the best 
> > > > > > >> or most efficient way to
> > > > > > implement it.
> > > > > > >> We have fuego setup in a server and all our continuous 
> > > > > > >> integration jobs (for multiple code bases) are setup in it.
> > > > > > >> But the issue is there are
> > > > > > too many builds happening, which puts pressure on the hard 
> > > > > > disk (so many read/writes) and also increases the docker container size (because of build logs).
> > > > > > >>
> > > > > > >> Hence I would like to know if we can distribute the load 
> > > > > > >> to another server, for e.g. job will be created on our 
> > > > > > >> main server but build will
> > > > > > happen on the agent on another server. The idea is to move 
> > > > > > out all cpu, memory, storage intensive activity to agents 
> > > > > > (other
> > > > > > servers) on the network. The resultant logs can also be 
> > > > > > stored on those servers
> > > > > (outside docker container) so that docker size won't be growing rapidly.
> > > > > > >>
> > > > > > >> I tried creating a new node and setup an agent (connect 
> > > > > > >> via
> > > > > > >> ssh) but facing issues in installing the 'SSH Build Agents'
> > > > > > >> plugin because of
> > > > > > compatibility issues (jenkins version needs to be upgraded).
> > > > > > >>
> > > > > > >> If anyone already tried and has a solution for this issue please let me know.
> > > > > > >>
> > > > > > >> Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)?
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Dhinakar K.
> > > > > > >> Samsung India, Bengaluru.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
> 
>