* RE[3]: Distribute continuous integration (build) to servers outside docker container [not found] <CGME20240315153759epcas5p3d49d4d0bfcf86c98f2c3754afa96034b@epcas5p3.samsung.com> @ 2024-03-15 15:37 ` dhinakar.k 2024-03-15 18:41 ` Bird, Tim 0 siblings, 1 reply; 4+ messages in thread From: dhinakar.k @ 2024-03-15 15:37 UTC (permalink / raw) To: 'Bird, Tim', fuego+owner, fuego Hi Tim, Thanks a lot for your response. Please find below the details that you requested. # du -sh /var/lib/Jenkins 32G /var/lib/Jenkins # du -sh /var/lib/jenkins/* | sort -h 0 /var/lib/jenkins/secret.key.not-so-secret 4.0K /var/lib/jenkins/com.cloudbees.hudson.plugins.folder.config.AbstractFolderConfiguration.xml 4.0K /var/lib/jenkins/com.mtvi.plateng.hudson.ldap.LdapMailAddressResolver.xml 4.0K /var/lib/jenkins/com.orctom.jenkins.plugin.buildtimestamp.BuildTimestampWrapper.xml 4.0K /var/lib/jenkins/credentials.xml 4.0K /var/lib/jenkins/hudson.maven.MavenModuleSet.xml 4.0K /var/lib/jenkins/hudson.model.UpdateCenter.xml 4.0K /var/lib/jenkins/hudson.plugins.emailext.ExtendedEmailPublisher.xml 4.0K /var/lib/jenkins/hudson.plugins.git.GitSCM.xml 4.0K /var/lib/jenkins/hudson.plugins.git.GitTool.xml 4.0K /var/lib/jenkins/hudson.plugins.sidebar_link.SidebarLinkPlugin.xml 4.0K /var/lib/jenkins/hudson.plugins.sonar.SonarGlobalConfiguration.xml 4.0K /var/lib/jenkins/hudson.plugins.timestamper.TimestamperConfig.xml 4.0K /var/lib/jenkins/hudson.tasks.Mailer.xml 4.0K /var/lib/jenkins/hudson.tasks.Shell.xml 4.0K /var/lib/jenkins/hudson.triggers.SCMTrigger.xml 4.0K /var/lib/jenkins/identity.key.enc 4.0K /var/lib/jenkins/io.jenkins.plugins.junit.storage.JunitTestResultStorageConfiguration.xml 4.0K /var/lib/jenkins/jenkins.fingerprints.GlobalFingerprintConfiguration.xml 4.0K /var/lib/jenkins/jenkins.install.InstallUtil.lastExecVersion 4.0K /var/lib/jenkins/jenkins.install.UpgradeWizard.state 4.0K /var/lib/jenkins/jenkins.model.ArtifactManagerConfiguration.xml 4.0K /var/lib/jenkins/jenkins.model.GlobalBuildDiscarderConfiguration.xml 4.0K /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml 4.0K /var/lib/jenkins/jenkins.plugins.msginject.MsgInjectConfig.xml 4.0K /var/lib/jenkins/jenkins.security.ResourceDomainConfiguration.xml 4.0K /var/lib/jenkins/jenkins.security.apitoken.ApiTokenPropertyConfiguration.xml 4.0K /var/lib/jenkins/jenkins.tasks.filters.EnvVarsFilterGlobalConfiguration.xml 4.0K /var/lib/jenkins/jenkins.telemetry.Correlator.xml 4.0K /var/lib/jenkins/net.plavcak.jenkins.plugins.scmskip.SCMSkipBuildWrapper.xml 4.0K /var/lib/jenkins/net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.xml 4.0K /var/lib/jenkins/nodeMonitors.xml 4.0K /var/lib/jenkins/org.codefirst.SimpleThemeDecorator.xml 4.0K /var/lib/jenkins/org.jenkinsCi.plugins.projectDescriptionSetter.DescriptionSetterWrapper.xml 4.0K /var/lib/jenkins/org.jenkinsci.plugins.ansible_tower.AnsibleTower.xml 4.0K /var/lib/jenkins/org.jenkinsci.plugins.corsfilter.AccessControlsFilter.xml 4.0K /var/lib/jenkins/org.jenkinsci.plugins.lucene.search.config.SearchBackendConfiguration.xml 4.0K /var/lib/jenkins/org.jenkinsci.plugins.workflow.flow.GlobalDefaultFlowDurabilityLevel.xml 4.0K /var/lib/jenkins/org.quality.gates.jenkins.plugin.GlobalConfig.xml 4.0K /var/lib/jenkins/queue.xml 4.0K /var/lib/jenkins/queue.xml.bak 4.0K /var/lib/jenkins/scriptApproval.xml 4.0K /var/lib/jenkins/secret.key 4.0K /var/lib/jenkins/sidebar-link.xml 12K /var/lib/jenkins/gerrit-trigger.xml 16K /var/lib/jenkins/hudson.plugins.ansicolor.AnsiColorBuildWrapper.xml 20K /var/lib/jenkins/config.xml 44K /var/lib/jenkins/nodes 72K /var/lib/jenkins/secrets 356K /var/lib/jenkins/users 1.7M /var/lib/jenkins/logs 1.8M /var/lib/jenkins/luceneIndex 3.6M /var/lib/jenkins/updates 3.9M /var/lib/jenkins/plugins_backup 316M /var/lib/jenkins/plugins 316M /var/lib/jenkins/plugins_not_working 372M /var/lib/jenkins/plugins_old_2ndFeb2024 376M /var/lib/jenkins/plugins_working_2ndFeb2024 1.8G /var/lib/jenkins/userContent 11G /var/lib/jenkins/jobs 19G /var/lib/jenkins/workspace # du -sh /fuego-rw 13G /fuego-rw # du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Dhrystone-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-aarch64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.IOzone-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Interbench-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Java-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.OpenSSL-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Stream-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Whetstone-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.bonnie-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.cyclictest-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.dbench4-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ebizzy-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ffsb-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.fio-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.gtkperf-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.hackbench-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.himeno-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.iperf-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.linpack-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.lmbench2-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.nbench_byte-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netperf-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netpipe-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.signaltest-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.tiobench-x86_64 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.x11perf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Dhrystone-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.GLMark-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.IOzone-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Interbench-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Java-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.OpenSSL-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Stream-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Whetstone-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.bonnie-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.cyclictest-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.dbench4-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ebizzy-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ffsb-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.fio-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.gtkperf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.hackbench-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.himeno-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.iperf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.linpack-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.lmbench2-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.nbench_byte-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netperf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netpipe-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.signaltest-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.tiobench-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.x11perf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.LTP-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.OpenSSL-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.aiostress-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bc-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bzip2-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.crashme-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.expat-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.fontconfig-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ft2demos-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.glib-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.hello_world-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ipv6connect-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.jpeg-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.linus_stress-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.netperf-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.pi_tests-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.rmaptest-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.scrashme-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.stress-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.synctest-x86_64 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.zlib-x86_64 4.0K /fuego-rw/buildzone/README 12K /fuego-rw/buildzone/Functional.bc-x86_64 32K /fuego-rw/buildzone/Functional.hello_world-x86_64 32K /fuego-rw/buildzone/Functional.linus_stress-x86_64 32K /fuego-rw/buildzone/Functional.synctest-x86_64 36K /fuego-rw/buildzone/Functional.rmaptest-x86_64 48K /fuego-rw/buildzone/Benchmark.himeno-x86_64 48K /fuego-rw/buildzone/Benchmark.linpack-x86_64 48K /fuego-rw/buildzone/Functional.ipv6connect-x86_64 60K /fuego-rw/buildzone/Benchmark.Stream-x86_64 76K /fuego-rw/buildzone/Benchmark.Whetstone-x86_64 76K /fuego-rw/buildzone/Benchmark.ebizzy-x86_64 84K /fuego-rw/buildzone/Functional.aiostress-x86_64 168K /fuego-rw/buildzone/Benchmark.Dhrystone-x86_64 240K /fuego-rw/buildzone/Benchmark.tiobench-x86_64 244K /fuego-rw/buildzone/Functional.crashme-x86_64 488K /fuego-rw/buildzone/Benchmark.Interbench-x86_64 576K /fuego-rw/buildzone/Functional.scrashme-x86_64 760K /fuego-rw/buildzone/Benchmark.hackbench-x86_64 816K /fuego-rw/buildzone/Benchmark.signaltest-x86_64 868K /fuego-rw/buildzone/Functional.pi_tests-x86_64 900K /fuego-rw/buildzone/Benchmark.netpipe-x86_64 904K /fuego-rw/buildzone/Benchmark.cyclictest-x86_64 1004K /fuego-rw/buildzone/Functional.stress-x86_64 1.3M /fuego-rw/buildzone/Benchmark.ffsb-x86_64 1.6M /fuego-rw/buildzone/Benchmark.GLMark-aarch64 1.6M /fuego-rw/buildzone/Benchmark.GLMark-x86_64 1.7M /fuego-rw/buildzone/Benchmark.nbench_byte-x86_64 2.1M /fuego-rw/buildzone/Benchmark.iperf-x86_64 2.3M /fuego-rw/buildzone/Benchmark.gtkperf-x86_64 2.4M /fuego-rw/buildzone/Benchmark.bonnie-x86_64 2.5M /fuego-rw/buildzone/Benchmark.x11perf-x86_64 2.9M /fuego-rw/buildzone/Functional.bzip2-x86_64 3.0M /fuego-rw/buildzone/Functional.zlib-x86_64 3.1M /fuego-rw/buildzone/Benchmark.IOzone-x86_64 3.5M /fuego-rw/buildzone/Functional.jpeg-x86_64 5.2M /fuego-rw/buildzone/Benchmark.lmbench2-x86_64 6.1M /fuego-rw/buildzone/Benchmark.netperf-x86_64 6.1M /fuego-rw/buildzone/Functional.netperf-x86_64 7.0M /fuego-rw/buildzone/Functional.fontconfig-x86_64 9.1M /fuego-rw/buildzone/Benchmark.fio-x86_64 29M /fuego-rw/buildzone/Functional.expat-x86_64 33M /fuego-rw/buildzone/Benchmark.dbench4-x86_64 34M /fuego-rw/buildzone/Functional.ft2demos-x86_64 49M /fuego-rw/buildzone/Functional.glib-x86_64 74M /fuego-rw/buildzone/Benchmark.OpenSSL-x86_64 74M /fuego-rw/buildzone/Functional.OpenSSL-x86_64 160M /fuego-rw/buildzone/Benchmark.Java-x86_64 1.8G /fuego-rw/buildzone/X86-SSH-Test.default.Functional.kernel_build-x86_64 2.4G /fuego-rw/buildzone/Functional.LTP-x86_64 Outside the container, can you run: $ docker image ls nxo-container-backup-2ndsep2021 latest b6a3d16508e8 2 years ago 173GB fuego-base-image-20thaug2019-container latest b2cf22891ac6 4 years ago 2.41GB fuegocontainerbackup-4thmar2019 latest 04112c383476 5 years ago 3.34GB fuego-piper-image-22ndnov2021 latest 0121b1c342e6 6 years ago 10.7GB fuego-xyz-image-22ndnov2021 latest 0bf6b942db70 6 years ago 5.8GB fuego-dart-image-22ndnov2021 latest 1d2c6a99dc0a 6 years ago 1.66GB fuego-pavv-image-22ndnov2021 latest ba71ede1e872 6 years ago 4.63GB fuego-gpu-image-22ndnov2021 latest 42718d86dd09 6 years ago 3.41GB Regarding below request, I just want to lock the Jenkins user interface/ dashboard, so that none can see all the jobs, results, logs and other details without logging in. So to start with even one user with login credentials will help. But I don't know how Jenkins credentials relate to user accounts. I haven't ever set up Jenkins to handle multiple user accounts. Indeed, in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to create or manage new user accounts. Thanks a lot for all other details, they were very helpful. Regards, Dhinakar K. -----Original Message----- From: Bird, Tim [mailto:Tim.Bird@sony.com] Sent: Wednesday, March 13, 2024 5:47 AM To: dhinakar.k <dhinakar.k@samsung.com>; fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org Subject: RE: HTML message rejected: Re: Distribute continuous integration (build) to servers outside docker container Hey Dhinakar, Sorry to not respond sooner. There was a lot to process in your email, and some of it I don't have answers for, but I'll try to give some status and feedback where I can. > -----Original Message----- > From: dhinakar.k <dhinakar.k@samsung.com> Hi Tim, > > I had sent a reply to your response on 3rd Feb.2024, not sure if it had reached you. > I am posting the same message below again. > > Thanks a lot for your prompt reply. > Iam looking forward to try the updated docker container with an > upgraded distribution base from Debian stretch to Debian bullseye, and from a Jenkins version of 2.249.3 to 2.414.1. Please let me whenever it is available. OK - Will do. I haven't been able to allocate time for testing and fixing it, so I didn't want to push it to the master branch. If I pushed something to a testing branch, would you be able to download from there and test it? One issue that came up is that some of the toolchains for some of my old boards/distros are not supported (ie don't work) in Debian bullseye, and so I was looking for solutions for that. > We are migrating to another server, so Iam looking for best options to > transfer all the content to the new server. Since the docker container > size is huge I want to trim it down by removing unwanted old logs. It would be good to know where the container size is coming from. Is it in the Fuego build directories, the Fuego run directories (where Fuego logs and run artifacts are stored), or in the Jenkins build directories (where Jenkins logs and artifacts are)? The materials under /fuego-rw should be on the host system, and not inside the container itself (since these are volume mounts). But I don't know how this space is accounted to the container (if any). It shouldn't be in any container image. The materials under /var/lib/jenkins are inside the container image. Can you run the following commands, inside your container, and send the results: # du -sh /var/lib/jenkins # du -sh /var/lib/jenkins/* | sort -h # du -sh /fuego-rw # du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l Outside the container, can you run: $ docker image ls (and remove any lines not related to fuego images) But I don't know how Jenkins credentials relate to user accounts. I haven't ever set up Jenkins to handle multiple user accounts. Indeed, in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to create or manage new user accounts. > Also, I am worried about server crash and hard disk crash due to so many read/write cycles (that happens during continuous integration). > Hence looking for the best way to back up the container and also to > distribute the load & read/write cycles to another server (via an ssh > agent). I tried with plain Jenkins via ssh credentials plugin, it seems feasible. Hence if fuego upgrades to debian bullseye it would be really helpful. OK - I'll keep plugging away at this. > > Also, would like to know, when login credentials support will be > available in fuego? Because as a security measure we are supposed to have login credentials for each user. Will be available after debian upgrade? I believe you are referring to Jenkins credentials? It appears that Jenkins supports credentials in version 2.414, which is the new version of Jenkins in the new docker container (with the Debian upgrade). So I believe the answer is yes. But I don't know how Jenkins credentials relate to user accounts. I haven't ever set up Jenkins to handle multiple user accounts. Indeed, in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to create or manage new user accounts. > With respect to removing the logs I will figure out the best way as I > see so many options but not sure if that will reduce the container size directly or need to run some container commands to trim it down. I will explore that as well and email my observations in this forum. I have considered adding an "rm-run" sub-command to ftc, which would allow you to remove old runs. However, I'd like to see if it's the run data that is causing the size problem that you are seeing. If you want to archive the data for an individual run, then you can use 'ftc package-run' and move the resulting package to some long-term storage device (before, ostensibly, removing that run from the system.) I hope this is helpful. -- Tim > > Regards, > Dhinakar K. > > -----Original Message----- > From: Bird, Tim [mailto:Tim.Bird@sony.com] > Sent: Saturday, February 3, 2024 3:44 AM > To: dhinakar k <dhinakar.k@gmail.com>; fuego+owner@lists.linux.dev; > fuego@lists.linuxfoundation.org > Subject: RE: HTML message rejected: Re: Distribute continuous > integration (build) to servers outside docker container > > Dhinakar, See my answers inline below.. > > > -----Original Message----- > > From: dhinakar k <dhinakar.k@gmail.com> Dear Fuego users, > > > > I have a requirement, so I thought of checking with the community if > > it was feasible and if so what is the best or most efficient way to > > implement it. > > We have fuego setup in a server and all our continuous integration > > jobs (for multiple code bases) are setup in it. But the issue is > > there are too many builds happening, which puts pressure on the hard > > disk (so many read/writes) and also increases the docker container > > size (because of build logs). > > > > Hence I would like to know if we can distribute the load to another > > server, for e.g. job will be created on our main server but build > > will happen on the agent on another server. The idea is to move out > > all cpu, memory, storage intensive activity to agents (other > > servers) on the network. The resultant logs can also be stored on > > those servers (outside docker container) so that docker size won't > > be growing rapidly. > > > > I tried creating a new node and setup an agent (connect via ssh) but > > facing issues in installing the 'SSH Build Agents' plugin because of > > compatibility issues (jenkins version needs to be upgraded). > > > > If anyone already tried and has a solution for this issue please let me know. > > Unfortunately, I have not tried this, so I'm not familiar with how Jenkins supports distributed jobs. > > But I have a few ideas to share. > > First, I'm working now on an update to the docker container, from a > distribution base of Debian stretch to Debian bullseye, and from a > Jenkins version of 2.249.3 to 2.414.1 (thanks to a patch provided by Fuego user Yoichi Tachibana). I have applied the patches, but found some issues with the build instructions for some packages working in Debian bullseye. So I have not committed the changes to > the Fuego master branch yet. I hope to be able to do this soon. > > This might help with the issue of not being able to use the ssh agent plugin for Jenkins. > > Second, it may be possible to do some kind of proxy job submission > thing, by creating proxy jobs on the main Jenkins server, but having the jobs actually executed ("built", in Jenkins terminology) on secondary servers. > > There are a number of ways of supporting remote execution, and how you > implement it will determine where the CPU cycles and memory usage is > during the job, and where the logs end up living (and taking up > space.) > > If you are using 'ttc' as your transport layer, recent versions of ttc > have support for remote boards via ssh (configured in ttc.conf). But that would leave logs on the master server. > > There are multiple ways to configure Fuego on the secondary servers - > for example using Docker containers or installing Fuego natively. In any event, it may be possible to set up proxy jobs on the main server that call 'ftc' on the secondary > servers, to run the jobs of interest. Depending on how this was structured > you may end up with build artifacts and logs on the secondary system, > and only Jenkins artifacts and logs on the main system (which should save space). > > It might be good to get a more detailed description of your Jenkins setup to be able to understand the options available. > > > > > Also, what is the best way to backup the container after trimming it > > down (remove unwanted files/logs etc. and reduce it's size)? > > I'm not sure how to backup a container in Docker. There's the 'docker container commit' > command, but I haven't used that. > > In general, to trim down you data usage you will want to perform a few operations. > Inside Jenkins, you can delete individual builds. You can likely > automate this (not have to do it manually through the UI), by using > the remote API. That should work for the material that is inside the container. Lots of data, however, including the build artifacts and the logs are natively on the host, and are only present in the container through bind mounts. > > This means you can remove this data directly using file operations (e.g. rm) on the host. > There are a few nuances to this, however, as some of the build directories are referenced via symlinks. > > 'ftc' currently supports removing jobs and nodes. I don't recall off the top of my head if this includes removing all the builds for a job or not. > But in any case, it sounds like you only want to remove some of the builds/runs, not the jobs themselves. > We could possibly support 'ftc rm-run', which could remove specific runs. 'ftc query-runs' > could be used to generate a list of runs (e.g. that were created before a certain date). > > Please provide a few more details, and maybe we can talk through and/or develop some solutions that will help meet your needs. > > -- Tim > > > > > On Thu, Feb 1, 2024 at 11:52 PM <fuego+owner@lists.linux.dev> wrote: > > > > > > Greetings! > > > > > > This is the mlmmj program managing the <fuego@lists.linux.dev> > > > mailing list. > > > > > > Your message to <fuego@lists.linux.dev> was not delivered to the > > > list because it contained a HTML part. Only text/plain messages > > > are allowed on this list. > > > > > > Please configure your mail client to only send plain text mail. > > > > > > For your reference, the rejected message follows below. > > > > > > > > > > > > ---------- Forwarded message ---------- > > > From: dhinakar k <dhinakar.k@gmail.com> > > > To: fuego+help@lists.linux.dev, fuego@lists.linuxfoundation.org, > > > "Bird, Timothy" <Tim.Bird@sony.com> > > > Cc: > > > Bcc: > > > Date: Thu, 1 Feb 2024 23:51:30 +0530 > > > Subject: Re: Distribute continuous integration (build) to servers > > > outside docker container Dear Fuego users, > > > > > > I have a requirement, so I thought of checking with the community > > > if it was feasible and if so what is the best or most efficient > > > way to > > implement it. > > > We have fuego setup in a server and all our continuous integration > > > jobs (for multiple code bases) are setup in it. But the issue is > > > there are > > too many builds happening, which puts pressure on the hard disk (so > > many read/writes) and also increases the docker container size (because of build logs). > > > > > > Hence I would like to know if we can distribute the load to > > > another server, for e.g. job will be created on our main server > > > but build will > > happen on the agent on another server. The idea is to move out all > > cpu, memory, storage intensive activity to agents (other servers) on > > the network. The resultant logs can also be stored on those servers > (outside docker container) so that docker size won't be growing rapidly. > > > > > > I tried creating a new node and setup an agent (connect via ssh) > > > but facing issues in installing the 'SSH Build Agents' plugin > > > because of > > compatibility issues (jenkins version needs to be upgraded). > > > > > > If anyone already tried and has a solution for this issue please let me know. > > > > > > Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)? > > > > > > Regards, > > > Dhinakar K. > > > Samsung India, Bengaluru. > > > > > > On Thu, Feb 1, 2024 at 10:30 PM dhinakar k <dhinakar.k@gmail.com> wrote: > > >> > > >> Dear Fuego users, > > >> > > >> I have a requirement, so I thought of checking with the community > > >> if it was feasible and if so what is the best or most efficient > > >> way to > > implement it. > > >> We have fuego setup in a server and all our continuous > > >> integration jobs (for multiple code bases) are setup in it. But > > >> the issue is there are > > too many builds happening, which puts pressure on the hard disk (so > > many read/writes) and also increases the docker container size (because of build logs). > > >> > > >> Hence I would like to know if we can distribute the load to > > >> another server, for e.g. job will be created on our main server > > >> but build will > > happen on the agent on another server. The idea is to move out all > > cpu, memory, storage intensive activity to agents (other servers) on > > the network. The resultant logs can also be stored on those servers > (outside docker container) so that docker size won't be growing rapidly. > > >> > > >> I tried creating a new node and setup an agent (connect via ssh) > > >> but facing issues in installing the 'SSH Build Agents' plugin > > >> because of > > compatibility issues (jenkins version needs to be upgraded). > > >> > > >> If anyone already tried and has a solution for this issue please let me know. > > >> > > >> Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)? > > >> > > >> Regards, > > >> Dhinakar K. > > >> Samsung India, Bengaluru. > > > ^ permalink raw reply [flat|nested] 4+ messages in thread
* RE: RE[3]: Distribute continuous integration (build) to servers outside docker container 2024-03-15 15:37 ` RE[3]: Distribute continuous integration (build) to servers outside docker container dhinakar.k @ 2024-03-15 18:41 ` Bird, Tim 2024-03-17 14:27 ` dhinakar.k 0 siblings, 1 reply; 4+ messages in thread From: Bird, Tim @ 2024-03-15 18:41 UTC (permalink / raw) To: dhinakar.k, fuego+owner@lists.linux.dev, fuego@lists.linuxfoundation.org Dhinkar, Can you send me the info for these as well: # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l -- Tim > -----Original Message----- > From: dhinakar.k <dhinakar.k@samsung.com> > Sent: Friday, March 15, 2024 9:38 AM > To: Bird, Tim <Tim.Bird@sony.com>; fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org > Subject: RE[3]: Distribute continuous integration (build) to servers outside docker container > > Hi Tim, Thanks a lot for your response. Please find below the details that you requested. # du -sh /var/lib/Jenkins 32G /var/lib/Jenkins # du -sh > /var/lib/jenkins/* | sort -h 0 /var/lib/jenkins/secret. key. not-so-secret 4. 0K > /var/lib/jenkins/com. cloudbees. hudson. plugins. folder. config. AbstractFolderConfiguration. xml > ZjQcmQRYFpfptBannerStart > Caution : This email originated from outside of Sony. > Do not click links or open any attachments unless you recognize the sender and know the content is safe. Please report phishing if unsure. > > ZjQcmQRYFpfptBannerEnd > Hi Tim, > > Thanks a lot for your response. > Please find below the details that you requested. > > # du -sh /var/lib/Jenkins > 32G /var/lib/Jenkins > > # du -sh /var/lib/jenkins/* | sort -h > 0 /var/lib/jenkins/secret.key.not-so-secret > 4.0K /var/lib/jenkins/com.cloudbees.hudson.plugins.folder.config.AbstractFolderConfiguration.xml > 4.0K /var/lib/jenkins/com.mtvi.plateng.hudson.ldap.LdapMailAddressResolver.xml > 4.0K /var/lib/jenkins/com.orctom.jenkins.plugin.buildtimestamp.BuildTimestampWrapper.xml > 4.0K /var/lib/jenkins/credentials.xml > 4.0K /var/lib/jenkins/hudson.maven.MavenModuleSet.xml > 4.0K /var/lib/jenkins/hudson.model.UpdateCenter.xml > 4.0K /var/lib/jenkins/hudson.plugins.emailext.ExtendedEmailPublisher.xml > 4.0K /var/lib/jenkins/hudson.plugins.git.GitSCM.xml > 4.0K /var/lib/jenkins/hudson.plugins.git.GitTool.xml > 4.0K /var/lib/jenkins/hudson.plugins.sidebar_link.SidebarLinkPlugin.xml > 4.0K /var/lib/jenkins/hudson.plugins.sonar.SonarGlobalConfiguration.xml > 4.0K /var/lib/jenkins/hudson.plugins.timestamper.TimestamperConfig.xml > 4.0K /var/lib/jenkins/hudson.tasks.Mailer.xml > 4.0K /var/lib/jenkins/hudson.tasks.Shell.xml > 4.0K /var/lib/jenkins/hudson.triggers.SCMTrigger.xml > 4.0K /var/lib/jenkins/identity.key.enc > 4.0K /var/lib/jenkins/io.jenkins.plugins.junit.storage.JunitTestResultStorageConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.fingerprints.GlobalFingerprintConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.install.InstallUtil.lastExecVersion > 4.0K /var/lib/jenkins/jenkins.install.UpgradeWizard.state > 4.0K /var/lib/jenkins/jenkins.model.ArtifactManagerConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.model.GlobalBuildDiscarderConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.plugins.msginject.MsgInjectConfig.xml > 4.0K /var/lib/jenkins/jenkins.security.ResourceDomainConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.security.apitoken.ApiTokenPropertyConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.tasks.filters.EnvVarsFilterGlobalConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.telemetry.Correlator.xml > 4.0K /var/lib/jenkins/net.plavcak.jenkins.plugins.scmskip.SCMSkipBuildWrapper.xml > 4.0K /var/lib/jenkins/net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.xml > 4.0K /var/lib/jenkins/nodeMonitors.xml > 4.0K /var/lib/jenkins/org.codefirst.SimpleThemeDecorator.xml > 4.0K /var/lib/jenkins/org.jenkinsCi.plugins.projectDescriptionSetter.DescriptionSetterWrapper.xml > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.ansible_tower.AnsibleTower.xml > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.corsfilter.AccessControlsFilter.xml > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.lucene.search.config.SearchBackendConfiguration.xml > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.workflow.flow.GlobalDefaultFlowDurabilityLevel.xml > 4.0K /var/lib/jenkins/org.quality.gates.jenkins.plugin.GlobalConfig.xml > 4.0K /var/lib/jenkins/queue.xml > 4.0K /var/lib/jenkins/queue.xml.bak > 4.0K /var/lib/jenkins/scriptApproval.xml > 4.0K /var/lib/jenkins/secret.key > 4.0K /var/lib/jenkins/sidebar-link.xml > 12K /var/lib/jenkins/gerrit-trigger.xml > 16K /var/lib/jenkins/hudson.plugins.ansicolor.AnsiColorBuildWrapper.xml > 20K /var/lib/jenkins/config.xml > 44K /var/lib/jenkins/nodes > 72K /var/lib/jenkins/secrets > 356K /var/lib/jenkins/users > 1.7M /var/lib/jenkins/logs > 1.8M /var/lib/jenkins/luceneIndex > 3.6M /var/lib/jenkins/updates > 3.9M /var/lib/jenkins/plugins_backup > 316M /var/lib/jenkins/plugins > 316M /var/lib/jenkins/plugins_not_working > 372M /var/lib/jenkins/plugins_old_2ndFeb2024 > 376M /var/lib/jenkins/plugins_working_2ndFeb2024 > 1.8G /var/lib/jenkins/userContent > 11G /var/lib/jenkins/jobs > 19G /var/lib/jenkins/workspace > > # du -sh /fuego-rw > 13G /fuego-rw > > # du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Dhrystone-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-aarch64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.IOzone-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Interbench-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Java-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.OpenSSL-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Stream-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Whetstone-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.bonnie-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.cyclictest-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.dbench4-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ebizzy-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ffsb-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.fio-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.gtkperf-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.hackbench-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.himeno-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.iperf-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.linpack-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.lmbench2-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.nbench_byte-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netperf-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netpipe-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.signaltest-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.tiobench-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.x11perf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Dhrystone-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.GLMark-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.IOzone-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Interbench-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Java-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.OpenSSL-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Stream-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Whetstone-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.bonnie-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.cyclictest-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.dbench4-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ebizzy-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ffsb-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.fio-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.gtkperf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.hackbench-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.himeno-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.iperf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.linpack-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.lmbench2-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.nbench_byte-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netperf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netpipe-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.signaltest-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.tiobench-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.x11perf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.LTP-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.OpenSSL-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.aiostress-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bc-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bzip2-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.crashme-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.expat-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.fontconfig-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ft2demos-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.glib-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.hello_world-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ipv6connect-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.jpeg-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.linus_stress-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.netperf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.pi_tests-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.rmaptest-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.scrashme-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.stress-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.synctest-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.zlib-x86_64 > 4.0K /fuego-rw/buildzone/README > 12K /fuego-rw/buildzone/Functional.bc-x86_64 > 32K /fuego-rw/buildzone/Functional.hello_world-x86_64 > 32K /fuego-rw/buildzone/Functional.linus_stress-x86_64 > 32K /fuego-rw/buildzone/Functional.synctest-x86_64 > 36K /fuego-rw/buildzone/Functional.rmaptest-x86_64 > 48K /fuego-rw/buildzone/Benchmark.himeno-x86_64 > 48K /fuego-rw/buildzone/Benchmark.linpack-x86_64 > 48K /fuego-rw/buildzone/Functional.ipv6connect-x86_64 > 60K /fuego-rw/buildzone/Benchmark.Stream-x86_64 > 76K /fuego-rw/buildzone/Benchmark.Whetstone-x86_64 > 76K /fuego-rw/buildzone/Benchmark.ebizzy-x86_64 > 84K /fuego-rw/buildzone/Functional.aiostress-x86_64 > 168K /fuego-rw/buildzone/Benchmark.Dhrystone-x86_64 > 240K /fuego-rw/buildzone/Benchmark.tiobench-x86_64 > 244K /fuego-rw/buildzone/Functional.crashme-x86_64 > 488K /fuego-rw/buildzone/Benchmark.Interbench-x86_64 > 576K /fuego-rw/buildzone/Functional.scrashme-x86_64 > 760K /fuego-rw/buildzone/Benchmark.hackbench-x86_64 > 816K /fuego-rw/buildzone/Benchmark.signaltest-x86_64 > 868K /fuego-rw/buildzone/Functional.pi_tests-x86_64 > 900K /fuego-rw/buildzone/Benchmark.netpipe-x86_64 > 904K /fuego-rw/buildzone/Benchmark.cyclictest-x86_64 > 1004K /fuego-rw/buildzone/Functional.stress-x86_64 > 1.3M /fuego-rw/buildzone/Benchmark.ffsb-x86_64 > 1.6M /fuego-rw/buildzone/Benchmark.GLMark-aarch64 > 1.6M /fuego-rw/buildzone/Benchmark.GLMark-x86_64 > 1.7M /fuego-rw/buildzone/Benchmark.nbench_byte-x86_64 > 2.1M /fuego-rw/buildzone/Benchmark.iperf-x86_64 > 2.3M /fuego-rw/buildzone/Benchmark.gtkperf-x86_64 > 2.4M /fuego-rw/buildzone/Benchmark.bonnie-x86_64 > 2.5M /fuego-rw/buildzone/Benchmark.x11perf-x86_64 > 2.9M /fuego-rw/buildzone/Functional.bzip2-x86_64 > 3.0M /fuego-rw/buildzone/Functional.zlib-x86_64 > 3.1M /fuego-rw/buildzone/Benchmark.IOzone-x86_64 > 3.5M /fuego-rw/buildzone/Functional.jpeg-x86_64 > 5.2M /fuego-rw/buildzone/Benchmark.lmbench2-x86_64 > 6.1M /fuego-rw/buildzone/Benchmark.netperf-x86_64 > 6.1M /fuego-rw/buildzone/Functional.netperf-x86_64 > 7.0M /fuego-rw/buildzone/Functional.fontconfig-x86_64 > 9.1M /fuego-rw/buildzone/Benchmark.fio-x86_64 > 29M /fuego-rw/buildzone/Functional.expat-x86_64 > 33M /fuego-rw/buildzone/Benchmark.dbench4-x86_64 > 34M /fuego-rw/buildzone/Functional.ft2demos-x86_64 > 49M /fuego-rw/buildzone/Functional.glib-x86_64 > 74M /fuego-rw/buildzone/Benchmark.OpenSSL-x86_64 > 74M /fuego-rw/buildzone/Functional.OpenSSL-x86_64 > 160M /fuego-rw/buildzone/Benchmark.Java-x86_64 > 1.8G /fuego-rw/buildzone/X86-SSH-Test.default.Functional.kernel_build-x86_64 > 2.4G /fuego-rw/buildzone/Functional.LTP-x86_64 > > Outside the container, can you run: > $ docker image ls > nxo-container-backup-2ndsep2021 latest b6a3d16508e8 2 years ago 173GB > fuego-base-image-20thaug2019-container latest b2cf22891ac6 4 years ago 2.41GB > fuegocontainerbackup-4thmar2019 latest 04112c383476 5 years ago 3.34GB > fuego-piper-image-22ndnov2021 latest 0121b1c342e6 6 years ago 10.7GB > fuego-xyz-image-22ndnov2021 latest 0bf6b942db70 6 years ago 5.8GB > fuego-dart-image-22ndnov2021 latest 1d2c6a99dc0a 6 years ago 1.66GB > fuego-pavv-image-22ndnov2021 latest ba71ede1e872 6 years ago 4.63GB > fuego-gpu-image-22ndnov2021 latest 42718d86dd09 6 years ago 3.41GB > > Regarding below request, I just want to lock the Jenkins user interface/ dashboard, so that none can see all the jobs, results, logs and other > details without logging in. So to start with even one user with login credentials will help. > But I don't know how Jenkins credentials relate to user accounts. I haven't ever set up Jenkins to handle multiple user accounts. Indeed, in > my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to > create or manage new user accounts. > > Thanks a lot for all other details, they were very helpful. > > Regards, > Dhinakar K. > > -----Original Message----- > From: Bird, Tim [mailto:Tim.Bird@sony.com] > Sent: Wednesday, March 13, 2024 5:47 AM > To: dhinakar.k <dhinakar.k@samsung.com>; fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org > Subject: RE: HTML message rejected: Re: Distribute continuous integration (build) to servers outside docker container > > Hey Dhinakar, > > Sorry to not respond sooner. There was a lot to process in your email, and some of it I don't have answers for, but I'll try to give some status > and feedback where I can. > > > -----Original Message----- > > From: dhinakar.k <dhinakar.k@samsung.com> Hi Tim, > > > > I had sent a reply to your response on 3rd Feb.2024, not sure if it had reached you. > > I am posting the same message below again. > > > > Thanks a lot for your prompt reply. > > Iam looking forward to try the updated docker container with an > > upgraded distribution base from Debian stretch to Debian bullseye, and from a Jenkins version of 2.249.3 to 2.414.1. Please let me > whenever it is available. > > OK - Will do. I haven't been able to allocate time for testing and fixing it, so I didn't want to push it to the master branch. If I pushed > something to a testing branch, would you be able to download from there and test it? > > One issue that came up is that some of the toolchains for some of my old boards/distros are not supported (ie don't work) in Debian bullseye, > and so I was looking for solutions for that. > > > We are migrating to another server, so Iam looking for best options to > > transfer all the content to the new server. Since the docker container > > size is huge I want to trim it down by removing unwanted old logs. > > It would be good to know where the container size is coming from. Is it in the Fuego build directories, the Fuego run directories (where Fuego > logs and run artifacts are stored), or in the Jenkins build directories (where Jenkins logs and artifacts are)? The materials under /fuego-rw > should be on the host system, and not inside the container itself (since these are volume mounts). But I don't know how this space is > accounted to the container (if any). It shouldn't be in any container image. > > The materials under /var/lib/jenkins are inside the container image. > > Can you run the following commands, inside your container, and send the results: > > # du -sh /var/lib/jenkins > # du -sh /var/lib/jenkins/* | sort -h > # du -sh /fuego-rw > # du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l > > Outside the container, can you run: > $ docker image ls > (and remove any lines not related to fuego images) > > But I don't know how Jenkins credentials relate to user accounts. I haven't ever set up Jenkins to handle multiple user accounts. Indeed, in > my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to > create or manage new user accounts. > > > Also, I am worried about server crash and hard disk crash due to so many read/write cycles (that happens during continuous integration). > > Hence looking for the best way to back up the container and also to > > distribute the load & read/write cycles to another server (via an ssh > > agent). I tried with plain Jenkins via ssh credentials plugin, it seems feasible. Hence if fuego upgrades to debian bullseye it would be really > helpful. > OK - I'll keep plugging away at this. > > > > > Also, would like to know, when login credentials support will be > > available in fuego? Because as a security measure we are supposed to have login credentials for each user. Will be available after debian > upgrade? > > I believe you are referring to Jenkins credentials? It appears that Jenkins supports credentials in version 2.414, which is the new version of > Jenkins in the new docker container (with the Debian upgrade). So I believe the answer is yes. > > But I don't know how Jenkins credentials relate to user accounts. I haven't ever set up Jenkins to handle multiple user accounts. Indeed, in > my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to > create or manage new user accounts. > > > With respect to removing the logs I will figure out the best way as I > > see so many options but not sure if that will reduce the container size directly or need to run some container commands to trim it down. I > will explore that as well and email my observations in this forum. > > I have considered adding an "rm-run" sub-command to ftc, which would allow you to remove old runs. > However, I'd like to see if it's the run data that is causing the size problem that you are seeing. > > If you want to archive the data for an individual run, then you can use 'ftc package-run' and move the resulting package to some long-term > storage device (before, ostensibly, removing that run from the system.) > > I hope this is helpful. > -- Tim > > > > > > > Regards, > > Dhinakar K. > > > > -----Original Message----- > > From: Bird, Tim [mailto:Tim.Bird@sony.com] > > Sent: Saturday, February 3, 2024 3:44 AM > > To: dhinakar k <dhinakar.k@gmail.com>; fuego+owner@lists.linux.dev; > > fuego@lists.linuxfoundation.org > > Subject: RE: HTML message rejected: Re: Distribute continuous > > integration (build) to servers outside docker container > > > > Dhinakar, See my answers inline below.. > > > > > -----Original Message----- > > > From: dhinakar k <dhinakar.k@gmail.com> Dear Fuego users, > > > > > > I have a requirement, so I thought of checking with the community if > > > it was feasible and if so what is the best or most efficient way to > > > implement it. > > > We have fuego setup in a server and all our continuous integration > > > jobs (for multiple code bases) are setup in it. But the issue is > > > there are too many builds happening, which puts pressure on the hard > > > disk (so many read/writes) and also increases the docker container > > > size (because of build logs). > > > > > > Hence I would like to know if we can distribute the load to another > > > server, for e.g. job will be created on our main server but build > > > will happen on the agent on another server. The idea is to move out > > > all cpu, memory, storage intensive activity to agents (other > > > servers) on the network. The resultant logs can also be stored on > > > those servers (outside docker container) so that docker size won't > > > be growing rapidly. > > > > > > I tried creating a new node and setup an agent (connect via ssh) but > > > facing issues in installing the 'SSH Build Agents' plugin because of > > > compatibility issues (jenkins version needs to be upgraded). > > > > > > If anyone already tried and has a solution for this issue please let me know. > > > > Unfortunately, I have not tried this, so I'm not familiar with how Jenkins supports distributed jobs. > > > > But I have a few ideas to share. > > > > First, I'm working now on an update to the docker container, from a > > distribution base of Debian stretch to Debian bullseye, and from a > > Jenkins version of 2.249.3 to 2.414.1 (thanks to a patch provided by Fuego user Yoichi Tachibana). I have applied the patches, but found > some issues with the build instructions for some packages working in Debian bullseye. So I have not committed the changes to > > the Fuego master branch yet. I hope to be able to do this soon. > > > > This might help with the issue of not being able to use the ssh agent plugin for Jenkins. > > > > Second, it may be possible to do some kind of proxy job submission > > thing, by creating proxy jobs on the main Jenkins server, but having the jobs actually executed ("built", in Jenkins terminology) on secondary > servers. > > > > There are a number of ways of supporting remote execution, and how you > > implement it will determine where the CPU cycles and memory usage is > > during the job, and where the logs end up living (and taking up > > space.) > > > > If you are using 'ttc' as your transport layer, recent versions of ttc > > have support for remote boards via ssh (configured in ttc.conf). But that would leave logs on the master server. > > > > There are multiple ways to configure Fuego on the secondary servers - > > for example using Docker containers or installing Fuego natively. In any event, it may be possible to set up proxy jobs on the main server that > call 'ftc' on the secondary > > servers, to run the jobs of interest. Depending on how this was structured > > you may end up with build artifacts and logs on the secondary system, > > and only Jenkins artifacts and logs on the main system (which should save space). > > > > It might be good to get a more detailed description of your Jenkins setup to be able to understand the options available. > > > > > > > > Also, what is the best way to backup the container after trimming it > > > down (remove unwanted files/logs etc. and reduce it's size)? > > > > I'm not sure how to backup a container in Docker. There's the 'docker container commit' > > command, but I haven't used that. > > > > In general, to trim down you data usage you will want to perform a few operations. > > Inside Jenkins, you can delete individual builds. You can likely > > automate this (not have to do it manually through the UI), by using > > the remote API. That should work for the material that is inside the container. Lots of data, however, including the build artifacts and the > logs are natively on the host, and are only present in the container through bind mounts. > > > > This means you can remove this data directly using file operations (e.g. rm) on the host. > > There are a few nuances to this, however, as some of the build directories are referenced via symlinks. > > > > 'ftc' currently supports removing jobs and nodes. I don't recall off the top of my head if this includes removing all the builds for a job or not. > > But in any case, it sounds like you only want to remove some of the builds/runs, not the jobs themselves. > > We could possibly support 'ftc rm-run', which could remove specific runs. 'ftc query-runs' > > could be used to generate a list of runs (e.g. that were created before a certain date). > > > > Please provide a few more details, and maybe we can talk through and/or develop some solutions that will help meet your needs. > > > > -- Tim > > > > > > > > On Thu, Feb 1, 2024 at 11:52 PM <fuego+owner@lists.linux.dev> wrote: > > > > > > > > Greetings! > > > > > > > > This is the mlmmj program managing the <fuego@lists.linux.dev> > > > > mailing list. > > > > > > > > Your message to <fuego@lists.linux.dev> was not delivered to the > > > > list because it contained a HTML part. Only text/plain messages > > > > are allowed on this list. > > > > > > > > Please configure your mail client to only send plain text mail. > > > > > > > > For your reference, the rejected message follows below. > > > > > > > > > > > > > > > > ---------- Forwarded message ---------- > > > > From: dhinakar k <dhinakar.k@gmail.com> > > > > To: fuego+help@lists.linux.dev, fuego@lists.linuxfoundation.org, > > > > "Bird, Timothy" <Tim.Bird@sony.com> > > > > Cc: > > > > Bcc: > > > > Date: Thu, 1 Feb 2024 23:51:30 +0530 > > > > Subject: Re: Distribute continuous integration (build) to servers > > > > outside docker container Dear Fuego users, > > > > > > > > I have a requirement, so I thought of checking with the community > > > > if it was feasible and if so what is the best or most efficient > > > > way to > > > implement it. > > > > We have fuego setup in a server and all our continuous integration > > > > jobs (for multiple code bases) are setup in it. But the issue is > > > > there are > > > too many builds happening, which puts pressure on the hard disk (so > > > many read/writes) and also increases the docker container size (because of build logs). > > > > > > > > Hence I would like to know if we can distribute the load to > > > > another server, for e.g. job will be created on our main server > > > > but build will > > > happen on the agent on another server. The idea is to move out all > > > cpu, memory, storage intensive activity to agents (other servers) on > > > the network. The resultant logs can also be stored on those servers > > (outside docker container) so that docker size won't be growing rapidly. > > > > > > > > I tried creating a new node and setup an agent (connect via ssh) > > > > but facing issues in installing the 'SSH Build Agents' plugin > > > > because of > > > compatibility issues (jenkins version needs to be upgraded). > > > > > > > > If anyone already tried and has a solution for this issue please let me know. > > > > > > > > Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)? > > > > > > > > Regards, > > > > Dhinakar K. > > > > Samsung India, Bengaluru. > > > > > > > > On Thu, Feb 1, 2024 at 10:30 PM dhinakar k <dhinakar.k@gmail.com> wrote: > > > >> > > > >> Dear Fuego users, > > > >> > > > >> I have a requirement, so I thought of checking with the community > > > >> if it was feasible and if so what is the best or most efficient > > > >> way to > > > implement it. > > > >> We have fuego setup in a server and all our continuous > > > >> integration jobs (for multiple code bases) are setup in it. But > > > >> the issue is there are > > > too many builds happening, which puts pressure on the hard disk (so > > > many read/writes) and also increases the docker container size (because of build logs). > > > >> > > > >> Hence I would like to know if we can distribute the load to > > > >> another server, for e.g. job will be created on our main server > > > >> but build will > > > happen on the agent on another server. The idea is to move out all > > > cpu, memory, storage intensive activity to agents (other servers) on > > > the network. The resultant logs can also be stored on those servers > > (outside docker container) so that docker size won't be growing rapidly. > > > >> > > > >> I tried creating a new node and setup an agent (connect via ssh) > > > >> but facing issues in installing the 'SSH Build Agents' plugin > > > >> because of > > > compatibility issues (jenkins version needs to be upgraded). > > > >> > > > >> If anyone already tried and has a solution for this issue please let me know. > > > >> > > > >> Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)? > > > >> > > > >> Regards, > > > >> Dhinakar K. > > > >> Samsung India, Bengaluru. > > > > > > > > ^ permalink raw reply [flat|nested] 4+ messages in thread
* RE: RE[3]: Distribute continuous integration (build) to servers outside docker container 2024-03-15 18:41 ` Bird, Tim @ 2024-03-17 14:27 ` dhinakar.k 2024-03-22 22:28 ` Bird, Tim 0 siblings, 1 reply; 4+ messages in thread From: dhinakar.k @ 2024-03-17 14:27 UTC (permalink / raw) To: 'Bird, Tim', fuego+owner, fuego; +Cc: dhinakar.k Hi Tim, Thanks for your prompt response. Please find the details you requested below. # du -sh /fuego-rw/logs/* | sort -h 4.0K /fuego-rw/logs/README 8.0K /fuego-rw/logs/logruns 56K /fuego-rw/logs/Benchmark.gtkperf 60K /fuego-rw/logs/Functional.glib 96K /fuego-rw/logs/Benchmark.GLMark 228K /fuego-rw/logs/theme 284K /fuego-rw/logs/Benchmark.dbench4 284K /fuego-rw/logs/Benchmark.himeno 284K /fuego-rw/logs/Benchmark.iperf 284K /fuego-rw/logs/Benchmark.x11perf 288K /fuego-rw/logs/Benchmark.linpack 288K /fuego-rw/logs/Functional.aiostress 288K /fuego-rw/logs/Functional.bc 288K /fuego-rw/logs/Functional.crashme 288K /fuego-rw/logs/Functional.fontconfig 288K /fuego-rw/logs/Functional.hello_world 288K /fuego-rw/logs/Functional.stress 288K /fuego-rw/logs/Functional.synctest 288K /fuego-rw/logs/Functional.zlib 292K /fuego-rw/logs/Benchmark.Stream 292K /fuego-rw/logs/Benchmark.netperf 292K /fuego-rw/logs/Benchmark.tiobench 296K /fuego-rw/logs/Functional.ipv6connect 304K /fuego-rw/logs/Benchmark.nbench_byte 304K /fuego-rw/logs/Benchmark.netpipe 312K /fuego-rw/logs/Benchmark.fio 312K /fuego-rw/logs/Functional.scrashme 316K /fuego-rw/logs/Benchmark.Java 344K /fuego-rw/logs/Functional.bzip2 352K /fuego-rw/logs/Benchmark.IOzone 352K /fuego-rw/logs/Functional.jpeg 356K /fuego-rw/logs/Benchmark.OpenSSL 372K /fuego-rw/logs/Functional.netperf 396K /fuego-rw/logs/Functional.expat 400K /fuego-rw/logs/Benchmark.lmbench2 404K /fuego-rw/logs/Functional.kernel_build 424K /fuego-rw/logs/Functional.rmaptest 524K /fuego-rw/logs/image 536K /fuego-rw/logs/Benchmark.Interbench 552K /fuego-rw/logs/Benchmark.ebizzy 552K /fuego-rw/logs/Benchmark.signaltest 556K /fuego-rw/logs/Functional.pi_tests 588K /fuego-rw/logs/Benchmark.Whetstone 740K /fuego-rw/logs/Functional.ft2demos 808K /fuego-rw/logs/Benchmark.Dhrystone 1.1M /fuego-rw/logs/Benchmark.cyclictest 1.1M /fuego-rw/logs/Benchmark.ffsb 1.2M /fuego-rw/logs/Benchmark.bonnie 1.6M /fuego-rw/logs/Benchmark.hackbench 2.1M /fuego-rw/logs/Functional.OpenSSL 7.5M /fuego-rw/logs/Extract 11M /fuego-rw/logs/Functional.linus_stress 12M /fuego-rw/logs/Functional.LTP # ftc list-nodes | wc -l 6 Jenkins nodes in this system: ARM64-SSH-Test ARM64-Serial-Test SAPIENT-MAIN-Server-Node X86-SSH-Test X86-Serial-Test # ftc list-jobs | wc -l 135 # ftc list-runs | wc -l 78 Regards, Dhinakar K. -----Original Message----- From: Bird, Tim [mailto:Tim.Bird@sony.com] Sent: Saturday, March 16, 2024 12:12 AM To: dhinakar.k <dhinakar.k@samsung.com>; fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org Subject: RE: RE[3]: Distribute continuous integration (build) to servers outside docker container Dhinkar, Can you send me the info for these as well: # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l -- Tim > -----Original Message----- > From: dhinakar.k <dhinakar.k@samsung.com> > Sent: Friday, March 15, 2024 9:38 AM > To: Bird, Tim <Tim.Bird@sony.com>; fuego+owner@lists.linux.dev; > fuego@lists.linuxfoundation.org > Subject: RE[3]: Distribute continuous integration (build) to servers > outside docker container > > Hi Tim, Thanks a lot for your response. Please find below the details > that you requested. # du -sh /var/lib/Jenkins 32G /var/lib/Jenkins # > du -sh > /var/lib/jenkins/* | sort -h 0 /var/lib/jenkins/secret. key. > not-so-secret 4. 0K /var/lib/jenkins/com. cloudbees. hudson. plugins. > folder. config. AbstractFolderConfiguration. xml > ZjQcmQRYFpfptBannerStart Caution : This email originated from outside of Sony. > Do not click links or open any attachments unless you recognize the sender and know the content is safe. Please report phishing if unsure. > > ZjQcmQRYFpfptBannerEnd > Hi Tim, > > Thanks a lot for your response. > Please find below the details that you requested. > > # du -sh /var/lib/Jenkins > 32G /var/lib/Jenkins > > # du -sh /var/lib/jenkins/* | sort -h > 0 /var/lib/jenkins/secret.key.not-so-secret > 4.0K /var/lib/jenkins/com.cloudbees.hudson.plugins.folder.config.AbstractFolderConfiguration.xml > 4.0K /var/lib/jenkins/com.mtvi.plateng.hudson.ldap.LdapMailAddressResolver.xml > 4.0K /var/lib/jenkins/com.orctom.jenkins.plugin.buildtimestamp.BuildTimestampWrapper.xml > 4.0K /var/lib/jenkins/credentials.xml > 4.0K /var/lib/jenkins/hudson.maven.MavenModuleSet.xml > 4.0K /var/lib/jenkins/hudson.model.UpdateCenter.xml > 4.0K /var/lib/jenkins/hudson.plugins.emailext.ExtendedEmailPublisher.xml > 4.0K /var/lib/jenkins/hudson.plugins.git.GitSCM.xml > 4.0K /var/lib/jenkins/hudson.plugins.git.GitTool.xml > 4.0K /var/lib/jenkins/hudson.plugins.sidebar_link.SidebarLinkPlugin.xml > 4.0K /var/lib/jenkins/hudson.plugins.sonar.SonarGlobalConfiguration.xml > 4.0K /var/lib/jenkins/hudson.plugins.timestamper.TimestamperConfig.xml > 4.0K /var/lib/jenkins/hudson.tasks.Mailer.xml > 4.0K /var/lib/jenkins/hudson.tasks.Shell.xml > 4.0K /var/lib/jenkins/hudson.triggers.SCMTrigger.xml > 4.0K /var/lib/jenkins/identity.key.enc > 4.0K /var/lib/jenkins/io.jenkins.plugins.junit.storage.JunitTestResultStorageConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.fingerprints.GlobalFingerprintConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.install.InstallUtil.lastExecVersion > 4.0K /var/lib/jenkins/jenkins.install.UpgradeWizard.state > 4.0K /var/lib/jenkins/jenkins.model.ArtifactManagerConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.model.GlobalBuildDiscarderConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.plugins.msginject.MsgInjectConfig.xml > 4.0K /var/lib/jenkins/jenkins.security.ResourceDomainConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.security.apitoken.ApiTokenPropertyConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.tasks.filters.EnvVarsFilterGlobalConfiguration.xml > 4.0K /var/lib/jenkins/jenkins.telemetry.Correlator.xml > 4.0K /var/lib/jenkins/net.plavcak.jenkins.plugins.scmskip.SCMSkipBuildWrapper.xml > 4.0K /var/lib/jenkins/net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.xml > 4.0K /var/lib/jenkins/nodeMonitors.xml > 4.0K /var/lib/jenkins/org.codefirst.SimpleThemeDecorator.xml > 4.0K /var/lib/jenkins/org.jenkinsCi.plugins.projectDescriptionSetter.DescriptionSetterWrapper.xml > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.ansible_tower.AnsibleTower.xml > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.corsfilter.AccessControlsFilter.xml > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.lucene.search.config.SearchBackendConfiguration.xml > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.workflow.flow.GlobalDefaultFlowDurabilityLevel.xml > 4.0K /var/lib/jenkins/org.quality.gates.jenkins.plugin.GlobalConfig.xml > 4.0K /var/lib/jenkins/queue.xml > 4.0K /var/lib/jenkins/queue.xml.bak > 4.0K /var/lib/jenkins/scriptApproval.xml > 4.0K /var/lib/jenkins/secret.key > 4.0K /var/lib/jenkins/sidebar-link.xml > 12K /var/lib/jenkins/gerrit-trigger.xml > 16K /var/lib/jenkins/hudson.plugins.ansicolor.AnsiColorBuildWrapper.xml > 20K /var/lib/jenkins/config.xml > 44K /var/lib/jenkins/nodes > 72K /var/lib/jenkins/secrets > 356K /var/lib/jenkins/users > 1.7M /var/lib/jenkins/logs > 1.8M /var/lib/jenkins/luceneIndex > 3.6M /var/lib/jenkins/updates > 3.9M /var/lib/jenkins/plugins_backup > 316M /var/lib/jenkins/plugins > 316M /var/lib/jenkins/plugins_not_working > 372M /var/lib/jenkins/plugins_old_2ndFeb2024 > 376M /var/lib/jenkins/plugins_working_2ndFeb2024 > 1.8G /var/lib/jenkins/userContent > 11G /var/lib/jenkins/jobs > 19G /var/lib/jenkins/workspace > > # du -sh /fuego-rw > 13G /fuego-rw > > # du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Dhrystone-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-aarch64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.IOzone-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Interbench-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Java-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.OpenSSL-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Stream-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Whetstone-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.bonnie-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.cyclictest-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.dbench4-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ebizzy-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ffsb-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.fio-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.gtkperf-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.hackbench-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.himeno-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.iperf-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.linpack-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.lmbench2-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.nbench_byte-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netperf-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netpipe-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.signaltest-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.tiobench-x86_64 > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.x11perf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Dhrystone-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.GLMark-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.IOzone-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Interbench-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Java-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.OpenSSL-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Stream-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Whetstone-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.bonnie-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.cyclictest-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.dbench4-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ebizzy-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ffsb-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.fio-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.gtkperf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.hackbench-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.himeno-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.iperf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.linpack-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.lmbench2-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.nbench_byte-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netperf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netpipe-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.signaltest-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.tiobench-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.x11perf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.LTP-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.OpenSSL-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.aiostress-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bc-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bzip2-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.crashme-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.expat-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.fontconfig-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ft2demos-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.glib-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.hello_world-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ipv6connect-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.jpeg-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.linus_stress-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.netperf-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.pi_tests-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.rmaptest-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.scrashme-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.stress-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.synctest-x86_64 > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.zlib-x86_64 > 4.0K /fuego-rw/buildzone/README > 12K /fuego-rw/buildzone/Functional.bc-x86_64 > 32K /fuego-rw/buildzone/Functional.hello_world-x86_64 > 32K /fuego-rw/buildzone/Functional.linus_stress-x86_64 > 32K /fuego-rw/buildzone/Functional.synctest-x86_64 > 36K /fuego-rw/buildzone/Functional.rmaptest-x86_64 > 48K /fuego-rw/buildzone/Benchmark.himeno-x86_64 > 48K /fuego-rw/buildzone/Benchmark.linpack-x86_64 > 48K /fuego-rw/buildzone/Functional.ipv6connect-x86_64 > 60K /fuego-rw/buildzone/Benchmark.Stream-x86_64 > 76K /fuego-rw/buildzone/Benchmark.Whetstone-x86_64 > 76K /fuego-rw/buildzone/Benchmark.ebizzy-x86_64 > 84K /fuego-rw/buildzone/Functional.aiostress-x86_64 > 168K /fuego-rw/buildzone/Benchmark.Dhrystone-x86_64 > 240K /fuego-rw/buildzone/Benchmark.tiobench-x86_64 > 244K /fuego-rw/buildzone/Functional.crashme-x86_64 > 488K /fuego-rw/buildzone/Benchmark.Interbench-x86_64 > 576K /fuego-rw/buildzone/Functional.scrashme-x86_64 > 760K /fuego-rw/buildzone/Benchmark.hackbench-x86_64 > 816K /fuego-rw/buildzone/Benchmark.signaltest-x86_64 > 868K /fuego-rw/buildzone/Functional.pi_tests-x86_64 > 900K /fuego-rw/buildzone/Benchmark.netpipe-x86_64 > 904K /fuego-rw/buildzone/Benchmark.cyclictest-x86_64 > 1004K /fuego-rw/buildzone/Functional.stress-x86_64 > 1.3M /fuego-rw/buildzone/Benchmark.ffsb-x86_64 > 1.6M /fuego-rw/buildzone/Benchmark.GLMark-aarch64 > 1.6M /fuego-rw/buildzone/Benchmark.GLMark-x86_64 > 1.7M /fuego-rw/buildzone/Benchmark.nbench_byte-x86_64 > 2.1M /fuego-rw/buildzone/Benchmark.iperf-x86_64 > 2.3M /fuego-rw/buildzone/Benchmark.gtkperf-x86_64 > 2.4M /fuego-rw/buildzone/Benchmark.bonnie-x86_64 > 2.5M /fuego-rw/buildzone/Benchmark.x11perf-x86_64 > 2.9M /fuego-rw/buildzone/Functional.bzip2-x86_64 > 3.0M /fuego-rw/buildzone/Functional.zlib-x86_64 > 3.1M /fuego-rw/buildzone/Benchmark.IOzone-x86_64 > 3.5M /fuego-rw/buildzone/Functional.jpeg-x86_64 > 5.2M /fuego-rw/buildzone/Benchmark.lmbench2-x86_64 > 6.1M /fuego-rw/buildzone/Benchmark.netperf-x86_64 > 6.1M /fuego-rw/buildzone/Functional.netperf-x86_64 > 7.0M /fuego-rw/buildzone/Functional.fontconfig-x86_64 > 9.1M /fuego-rw/buildzone/Benchmark.fio-x86_64 > 29M /fuego-rw/buildzone/Functional.expat-x86_64 > 33M /fuego-rw/buildzone/Benchmark.dbench4-x86_64 > 34M /fuego-rw/buildzone/Functional.ft2demos-x86_64 > 49M /fuego-rw/buildzone/Functional.glib-x86_64 > 74M /fuego-rw/buildzone/Benchmark.OpenSSL-x86_64 > 74M /fuego-rw/buildzone/Functional.OpenSSL-x86_64 > 160M /fuego-rw/buildzone/Benchmark.Java-x86_64 > 1.8G /fuego-rw/buildzone/X86-SSH-Test.default.Functional.kernel_build-x86_64 > 2.4G /fuego-rw/buildzone/Functional.LTP-x86_64 > > Outside the container, can you run: > $ docker image ls > nxo-container-backup-2ndsep2021 latest b6a3d16508e8 2 years ago 173GB > fuego-base-image-20thaug2019-container latest b2cf22891ac6 4 years ago 2.41GB > fuegocontainerbackup-4thmar2019 latest 04112c383476 5 years ago 3.34GB > fuego-piper-image-22ndnov2021 latest 0121b1c342e6 6 years ago 10.7GB > fuego-xyz-image-22ndnov2021 latest 0bf6b942db70 6 years ago 5.8GB > fuego-dart-image-22ndnov2021 latest 1d2c6a99dc0a 6 years ago 1.66GB > fuego-pavv-image-22ndnov2021 latest ba71ede1e872 6 years ago 4.63GB > fuego-gpu-image-22ndnov2021 latest 42718d86dd09 6 years ago 3.41GB > > Regarding below request, I just want to lock the Jenkins user > interface/ dashboard, so that none can see all the jobs, results, logs and other details without logging in. So to start with even one user with login credentials will help. > But I don't know how Jenkins credentials relate to user accounts. I > haven't ever set up Jenkins to handle multiple user accounts. Indeed, > in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to create or manage new user accounts. > > Thanks a lot for all other details, they were very helpful. > > Regards, > Dhinakar K. > > -----Original Message----- > From: Bird, Tim [mailto:Tim.Bird@sony.com] > Sent: Wednesday, March 13, 2024 5:47 AM > To: dhinakar.k <dhinakar.k@samsung.com>; fuego+owner@lists.linux.dev; > fuego@lists.linuxfoundation.org > Subject: RE: HTML message rejected: Re: Distribute continuous > integration (build) to servers outside docker container > > Hey Dhinakar, > > Sorry to not respond sooner. There was a lot to process in your > email, and some of it I don't have answers for, but I'll try to give some status and feedback where I can. > > > -----Original Message----- > > From: dhinakar.k <dhinakar.k@samsung.com> Hi Tim, > > > > I had sent a reply to your response on 3rd Feb.2024, not sure if it had reached you. > > I am posting the same message below again. > > > > Thanks a lot for your prompt reply. > > Iam looking forward to try the updated docker container with an > > upgraded distribution base from Debian stretch to Debian bullseye, > > and from a Jenkins version of 2.249.3 to 2.414.1. Please let me > whenever it is available. > > OK - Will do. I haven't been able to allocate time for testing and > fixing it, so I didn't want to push it to the master branch. If I pushed something to a testing branch, would you be able to download from there and test it? > > One issue that came up is that some of the toolchains for some of my > old boards/distros are not supported (ie don't work) in Debian bullseye, and so I was looking for solutions for that. > > > We are migrating to another server, so Iam looking for best options > > to transfer all the content to the new server. Since the docker > > container size is huge I want to trim it down by removing unwanted old logs. > > It would be good to know where the container size is coming from. Is > it in the Fuego build directories, the Fuego run directories (where > Fuego logs and run artifacts are stored), or in the Jenkins build > directories (where Jenkins logs and artifacts are)? The materials under /fuego-rw should be on the host system, and not inside the container itself (since these are volume mounts). But I don't know how this space is accounted to the container (if any). It shouldn't be in any container image. > > The materials under /var/lib/jenkins are inside the container image. > > Can you run the following commands, inside your container, and send the results: > > # du -sh /var/lib/jenkins > # du -sh /var/lib/jenkins/* | sort -h > # du -sh /fuego-rw > # du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | > sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc > list-runs | wc -l > > Outside the container, can you run: > $ docker image ls > (and remove any lines not related to fuego images) > > But I don't know how Jenkins credentials relate to user accounts. I > haven't ever set up Jenkins to handle multiple user accounts. Indeed, > in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to create or manage new user accounts. > > > Also, I am worried about server crash and hard disk crash due to so many read/write cycles (that happens during continuous integration). > > Hence looking for the best way to back up the container and also to > > distribute the load & read/write cycles to another server (via an > > ssh agent). I tried with plain Jenkins via ssh credentials plugin, > > it seems feasible. Hence if fuego upgrades to debian bullseye it > > would be really > helpful. > OK - I'll keep plugging away at this. > > > > > Also, would like to know, when login credentials support will be > > available in fuego? Because as a security measure we are supposed to > > have login credentials for each user. Will be available after debian > upgrade? > > I believe you are referring to Jenkins credentials? It appears that > Jenkins supports credentials in version 2.414, which is the new version of Jenkins in the new docker container (with the Debian upgrade). So I believe the answer is yes. > > But I don't know how Jenkins credentials relate to user accounts. I > haven't ever set up Jenkins to handle multiple user accounts. Indeed, > in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way to create or manage new user accounts. > > > With respect to removing the logs I will figure out the best way as > > I see so many options but not sure if that will reduce the container > > size directly or need to run some container commands to trim it > > down. I > will explore that as well and email my observations in this forum. > > I have considered adding an "rm-run" sub-command to ftc, which would allow you to remove old runs. > However, I'd like to see if it's the run data that is causing the size problem that you are seeing. > > If you want to archive the data for an individual run, then you can > use 'ftc package-run' and move the resulting package to some long-term > storage device (before, ostensibly, removing that run from the > system.) > > I hope this is helpful. > -- Tim > > > > > > > Regards, > > Dhinakar K. > > > > -----Original Message----- > > From: Bird, Tim [mailto:Tim.Bird@sony.com] > > Sent: Saturday, February 3, 2024 3:44 AM > > To: dhinakar k <dhinakar.k@gmail.com>; fuego+owner@lists.linux.dev; > > fuego@lists.linuxfoundation.org > > Subject: RE: HTML message rejected: Re: Distribute continuous > > integration (build) to servers outside docker container > > > > Dhinakar, See my answers inline below.. > > > > > -----Original Message----- > > > From: dhinakar k <dhinakar.k@gmail.com> Dear Fuego users, > > > > > > I have a requirement, so I thought of checking with the community > > > if it was feasible and if so what is the best or most efficient > > > way to implement it. > > > We have fuego setup in a server and all our continuous integration > > > jobs (for multiple code bases) are setup in it. But the issue is > > > there are too many builds happening, which puts pressure on the > > > hard disk (so many read/writes) and also increases the docker > > > container size (because of build logs). > > > > > > Hence I would like to know if we can distribute the load to > > > another server, for e.g. job will be created on our main server > > > but build will happen on the agent on another server. The idea is > > > to move out all cpu, memory, storage intensive activity to agents > > > (other > > > servers) on the network. The resultant logs can also be stored on > > > those servers (outside docker container) so that docker size won't > > > be growing rapidly. > > > > > > I tried creating a new node and setup an agent (connect via ssh) > > > but facing issues in installing the 'SSH Build Agents' plugin > > > because of compatibility issues (jenkins version needs to be upgraded). > > > > > > If anyone already tried and has a solution for this issue please let me know. > > > > Unfortunately, I have not tried this, so I'm not familiar with how Jenkins supports distributed jobs. > > > > But I have a few ideas to share. > > > > First, I'm working now on an update to the docker container, from a > > distribution base of Debian stretch to Debian bullseye, and from a > > Jenkins version of 2.249.3 to 2.414.1 (thanks to a patch provided by > > Fuego user Yoichi Tachibana). I have applied the patches, but found > some issues with the build instructions for some packages working in > Debian bullseye. So I have not committed the changes to > > the Fuego master branch yet. I hope to be able to do this soon. > > > > This might help with the issue of not being able to use the ssh agent plugin for Jenkins. > > > > Second, it may be possible to do some kind of proxy job submission > > thing, by creating proxy jobs on the main Jenkins server, but having > > the jobs actually executed ("built", in Jenkins terminology) on > > secondary > servers. > > > > There are a number of ways of supporting remote execution, and how > > you implement it will determine where the CPU cycles and memory > > usage is during the job, and where the logs end up living (and > > taking up > > space.) > > > > If you are using 'ttc' as your transport layer, recent versions of > > ttc have support for remote boards via ssh (configured in ttc.conf). But that would leave logs on the master server. > > > > There are multiple ways to configure Fuego on the secondary servers > > - for example using Docker containers or installing Fuego natively. > > In any event, it may be possible to set up proxy jobs on the main > > server that > call 'ftc' on the secondary > > servers, to run the jobs of interest. Depending on how this was structured > > you may end up with build artifacts and logs on the secondary > > system, and only Jenkins artifacts and logs on the main system (which should save space). > > > > It might be good to get a more detailed description of your Jenkins setup to be able to understand the options available. > > > > > > > > Also, what is the best way to backup the container after trimming > > > it down (remove unwanted files/logs etc. and reduce it's size)? > > > > I'm not sure how to backup a container in Docker. There's the 'docker container commit' > > command, but I haven't used that. > > > > In general, to trim down you data usage you will want to perform a few operations. > > Inside Jenkins, you can delete individual builds. You can likely > > automate this (not have to do it manually through the UI), by using > > the remote API. That should work for the material that is inside > > the container. Lots of data, however, including the build artifacts > > and the > logs are natively on the host, and are only present in the container through bind mounts. > > > > This means you can remove this data directly using file operations (e.g. rm) on the host. > > There are a few nuances to this, however, as some of the build directories are referenced via symlinks. > > > > 'ftc' currently supports removing jobs and nodes. I don't recall off the top of my head if this includes removing all the builds for a job or not. > > But in any case, it sounds like you only want to remove some of the builds/runs, not the jobs themselves. > > We could possibly support 'ftc rm-run', which could remove specific runs. 'ftc query-runs' > > could be used to generate a list of runs (e.g. that were created before a certain date). > > > > Please provide a few more details, and maybe we can talk through and/or develop some solutions that will help meet your needs. > > > > -- Tim > > > > > > > > On Thu, Feb 1, 2024 at 11:52 PM <fuego+owner@lists.linux.dev> wrote: > > > > > > > > Greetings! > > > > > > > > This is the mlmmj program managing the <fuego@lists.linux.dev> > > > > mailing list. > > > > > > > > Your message to <fuego@lists.linux.dev> was not delivered to the > > > > list because it contained a HTML part. Only text/plain messages > > > > are allowed on this list. > > > > > > > > Please configure your mail client to only send plain text mail. > > > > > > > > For your reference, the rejected message follows below. > > > > > > > > > > > > > > > > ---------- Forwarded message ---------- > > > > From: dhinakar k <dhinakar.k@gmail.com> > > > > To: fuego+help@lists.linux.dev, fuego@lists.linuxfoundation.org, > > > > "Bird, Timothy" <Tim.Bird@sony.com> > > > > Cc: > > > > Bcc: > > > > Date: Thu, 1 Feb 2024 23:51:30 +0530 > > > > Subject: Re: Distribute continuous integration (build) to > > > > servers outside docker container Dear Fuego users, > > > > > > > > I have a requirement, so I thought of checking with the > > > > community if it was feasible and if so what is the best or most > > > > efficient way to > > > implement it. > > > > We have fuego setup in a server and all our continuous > > > > integration jobs (for multiple code bases) are setup in it. But > > > > the issue is there are > > > too many builds happening, which puts pressure on the hard disk > > > (so many read/writes) and also increases the docker container size (because of build logs). > > > > > > > > Hence I would like to know if we can distribute the load to > > > > another server, for e.g. job will be created on our main server > > > > but build will > > > happen on the agent on another server. The idea is to move out all > > > cpu, memory, storage intensive activity to agents (other servers) > > > on the network. The resultant logs can also be stored on those > > > servers > > (outside docker container) so that docker size won't be growing rapidly. > > > > > > > > I tried creating a new node and setup an agent (connect via ssh) > > > > but facing issues in installing the 'SSH Build Agents' plugin > > > > because of > > > compatibility issues (jenkins version needs to be upgraded). > > > > > > > > If anyone already tried and has a solution for this issue please let me know. > > > > > > > > Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)? > > > > > > > > Regards, > > > > Dhinakar K. > > > > Samsung India, Bengaluru. > > > > > > > > On Thu, Feb 1, 2024 at 10:30 PM dhinakar k <dhinakar.k@gmail.com> wrote: > > > >> > > > >> Dear Fuego users, > > > >> > > > >> I have a requirement, so I thought of checking with the > > > >> community if it was feasible and if so what is the best or most > > > >> efficient way to > > > implement it. > > > >> We have fuego setup in a server and all our continuous > > > >> integration jobs (for multiple code bases) are setup in it. But > > > >> the issue is there are > > > too many builds happening, which puts pressure on the hard disk > > > (so many read/writes) and also increases the docker container size (because of build logs). > > > >> > > > >> Hence I would like to know if we can distribute the load to > > > >> another server, for e.g. job will be created on our main server > > > >> but build will > > > happen on the agent on another server. The idea is to move out all > > > cpu, memory, storage intensive activity to agents (other servers) > > > on the network. The resultant logs can also be stored on those > > > servers > > (outside docker container) so that docker size won't be growing rapidly. > > > >> > > > >> I tried creating a new node and setup an agent (connect via > > > >> ssh) but facing issues in installing the 'SSH Build Agents' > > > >> plugin because of > > > compatibility issues (jenkins version needs to be upgraded). > > > >> > > > >> If anyone already tried and has a solution for this issue please let me know. > > > >> > > > >> Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)? > > > >> > > > >> Regards, > > > >> Dhinakar K. > > > >> Samsung India, Bengaluru. > > > > > > > > ^ permalink raw reply [flat|nested] 4+ messages in thread
* RE: RE[3]: Distribute continuous integration (build) to servers outside docker container 2024-03-17 14:27 ` dhinakar.k @ 2024-03-22 22:28 ` Bird, Tim 0 siblings, 0 replies; 4+ messages in thread From: Bird, Tim @ 2024-03-22 22:28 UTC (permalink / raw) To: dhinakar.k, fuego+owner@lists.linux.dev, fuego@lists.linuxfoundation.org Dhinakar, OK - here's a summary of your object counts and storage issues: /var/lib/jenkins - 32G .../userContent - 1.8G .../jobs - 11G .../workspace - 19G /fuego-rw - 13G ../buildzone - ~6G ../buildzone/X86-SSH-Test.default.Functional.kernel_build-x86-64 - 1.8G ../buildzone/Functional.LTP-x86_64 - 2.4G 6 nodes, 135 jobs, 78 runs docker image: 2.41G? (or is it 173 GB?) This is confusing. Your latest fuego docker container is 2.41G, but you have /var/lib/jenkins that is about 32G (and that should all be in your container image). It looks like the builds for the kernel and for LTP are taking the most space. I have no idea why your /var/lib/jenkins/jobs directory would be so big (11G) Maybe do: $ du -sh /var/lib/jenkins/jobs/* | sort -sh and see which jobs have a lot of info. Also, I don't know what's going on with your /var/lib/jenkins/workspace (19G) I don't even have that directory on my system. My stats look quite different: I have 13 nodes, 387 jobs and 1129 runs in my most active Fuego container, my /var/lib/Jenkins is only 204M. My /fuego-rw/buildzone is 64G, due to lots of instances of LTP (and something very funky about one Interbench build): 1.2G ren1.latest-smoke.Functional.LTP-poky-aarch64 1.7G Functional.kselftest-x86_64 1.7G rpi3-2.default.Functional.kernel_build-aarch64 1.9G Functional.LTP_one_at_a_time-debian-armhf 2.0G Functional.raspi_kinstall-debian-armhf 2.3G bbb.latest-smoke.Functional.LTP-debian-armhf 2.4G Functional.LTP-x86_64 2.4G Functional.LTP_one_at_a_time-x86_64 2.5G Functional.LTP-poky-aarch64 2.5G Functional.LTP_one_at_a_time-aarch64 2.5G Functional.LTP_one_at_a_time-poky-aarch64 3.0G min1.latest-smoke.Functional.LTP-x86_64 3.0G min1.latest.Functional.LTP-x86_64 3.2G rpi3-2.latest-smoke.Functional.LTP-aarch64 32G Benchmark.Interbench-x86_64 There might be stuff in /var/lib/jenkins/workspace you could get rid of. I have a lot more runs on my system, but very many of those have very short run.json files, as they are for tests with small results. I believe the largest run.json and consolelog.txt files are for LTP. If you have a lot of those, maybe you can archive old ones to reduce disk usage and improve performance. In any event, is there any particular operation which is taking a long time to complete, due to the docker size (or fuego directory) being too big or taking too much space? Finally, I'm still working on the new docker container (with more recent versions of Debian and Jenkins). I've already fixed a few bugs that I found, but there are a lot of old Fuego packages that don't compile with newer toolchains. There are workarounds for this, but it will take a while to implement. My plan is to publish my working branch for this work on bitbucket, as a 'next' branch, even though not all problems are resolved. You could then build a container using that (if you want to), and check if the new version of Jenkins has the features you need for user authentication, security, and job distribution. I can give you some install steps for using a 'next' branch, if you are interested in trying that. Sorry this has taken so long. -- Tim > -----Original Message----- > From: dhinakar.k <dhinakar.k@samsung.com> > Hi Tim, > > Thanks for your prompt response. > Please find the details you requested below. > > # du -sh /fuego-rw/logs/* | sort -h > 4.0K /fuego-rw/logs/README > 8.0K /fuego-rw/logs/logruns > 56K /fuego-rw/logs/Benchmark.gtkperf > 60K /fuego-rw/logs/Functional.glib > 96K /fuego-rw/logs/Benchmark.GLMark > 228K /fuego-rw/logs/theme > 284K /fuego-rw/logs/Benchmark.dbench4 > 284K /fuego-rw/logs/Benchmark.himeno > 284K /fuego-rw/logs/Benchmark.iperf > 284K /fuego-rw/logs/Benchmark.x11perf > 288K /fuego-rw/logs/Benchmark.linpack > 288K /fuego-rw/logs/Functional.aiostress > 288K /fuego-rw/logs/Functional.bc > 288K /fuego-rw/logs/Functional.crashme > 288K /fuego-rw/logs/Functional.fontconfig > 288K /fuego-rw/logs/Functional.hello_world > 288K /fuego-rw/logs/Functional.stress > 288K /fuego-rw/logs/Functional.synctest > 288K /fuego-rw/logs/Functional.zlib > 292K /fuego-rw/logs/Benchmark.Stream > 292K /fuego-rw/logs/Benchmark.netperf > 292K /fuego-rw/logs/Benchmark.tiobench > 296K /fuego-rw/logs/Functional.ipv6connect > 304K /fuego-rw/logs/Benchmark.nbench_byte > 304K /fuego-rw/logs/Benchmark.netpipe > 312K /fuego-rw/logs/Benchmark.fio > 312K /fuego-rw/logs/Functional.scrashme > 316K /fuego-rw/logs/Benchmark.Java > 344K /fuego-rw/logs/Functional.bzip2 > 352K /fuego-rw/logs/Benchmark.IOzone > 352K /fuego-rw/logs/Functional.jpeg > 356K /fuego-rw/logs/Benchmark.OpenSSL > 372K /fuego-rw/logs/Functional.netperf > 396K /fuego-rw/logs/Functional.expat > 400K /fuego-rw/logs/Benchmark.lmbench2 > 404K /fuego-rw/logs/Functional.kernel_build > 424K /fuego-rw/logs/Functional.rmaptest > 524K /fuego-rw/logs/image > 536K /fuego-rw/logs/Benchmark.Interbench > 552K /fuego-rw/logs/Benchmark.ebizzy > 552K /fuego-rw/logs/Benchmark.signaltest > 556K /fuego-rw/logs/Functional.pi_tests > 588K /fuego-rw/logs/Benchmark.Whetstone > 740K /fuego-rw/logs/Functional.ft2demos > 808K /fuego-rw/logs/Benchmark.Dhrystone > 1.1M /fuego-rw/logs/Benchmark.cyclictest > 1.1M /fuego-rw/logs/Benchmark.ffsb > 1.2M /fuego-rw/logs/Benchmark.bonnie > 1.6M /fuego-rw/logs/Benchmark.hackbench > 2.1M /fuego-rw/logs/Functional.OpenSSL > 7.5M /fuego-rw/logs/Extract > 11M /fuego-rw/logs/Functional.linus_stress > 12M /fuego-rw/logs/Functional.LTP > > # ftc list-nodes | wc -l > 6 > > Jenkins nodes in this system: > ARM64-SSH-Test > ARM64-Serial-Test > SAPIENT-MAIN-Server-Node > X86-SSH-Test > X86-Serial-Test > > # ftc list-jobs | wc -l > 135 > > # ftc list-runs | wc -l > 78 > > Regards, > Dhinakar K. > > -----Original Message----- > From: Bird, Tim [mailto:Tim.Bird@sony.com] > Sent: Saturday, March 16, 2024 12:12 AM > To: dhinakar.k <dhinakar.k@samsung.com>; fuego+owner@lists.linux.dev; fuego@lists.linuxfoundation.org > Subject: RE: RE[3]: Distribute continuous integration (build) to servers outside docker container > > Dhinkar, > > Can you send me the info for these as well: > > # du -sh /fuego-rw/logs/* | sort -h > > > # ftc list-nodes | wc -l > > > # ftc list-jobs | wc -l > > > # ftc list-runs | wc -l > > > -- Tim > > > -----Original Message----- > > From: dhinakar.k <dhinakar.k@samsung.com> > > Sent: Friday, March 15, 2024 9:38 AM > > To: Bird, Tim <Tim.Bird@sony.com>; fuego+owner@lists.linux.dev; > > fuego@lists.linuxfoundation.org > > Subject: RE[3]: Distribute continuous integration (build) to servers > > outside docker container > > > > Hi Tim, Thanks a lot for your response. Please find below the details > > that you requested. # du -sh /var/lib/Jenkins 32G /var/lib/Jenkins # > > du -sh > > /var/lib/jenkins/* | sort -h 0 /var/lib/jenkins/secret. key. > > not-so-secret 4. 0K /var/lib/jenkins/com. cloudbees. hudson. plugins. > > folder. config. AbstractFolderConfiguration. xml > > ZjQcmQRYFpfptBannerStart Caution : This email originated from outside of Sony. > > Do not click links or open any attachments unless you recognize the sender and know the content is safe. Please report phishing if unsure. > > > > ZjQcmQRYFpfptBannerEnd > > Hi Tim, > > > > Thanks a lot for your response. > > Please find below the details that you requested. > > > > # du -sh /var/lib/Jenkins > > 32G /var/lib/Jenkins > > > > # du -sh /var/lib/jenkins/* | sort -h > > 0 /var/lib/jenkins/secret.key.not-so-secret > > 4.0K /var/lib/jenkins/com.cloudbees.hudson.plugins.folder.config.AbstractFolderConfiguration.xml > > 4.0K /var/lib/jenkins/com.mtvi.plateng.hudson.ldap.LdapMailAddressResolver.xml > > 4.0K /var/lib/jenkins/com.orctom.jenkins.plugin.buildtimestamp.BuildTimestampWrapper.xml > > 4.0K /var/lib/jenkins/credentials.xml > > 4.0K /var/lib/jenkins/hudson.maven.MavenModuleSet.xml > > 4.0K /var/lib/jenkins/hudson.model.UpdateCenter.xml > > 4.0K /var/lib/jenkins/hudson.plugins.emailext.ExtendedEmailPublisher.xml > > 4.0K /var/lib/jenkins/hudson.plugins.git.GitSCM.xml > > 4.0K /var/lib/jenkins/hudson.plugins.git.GitTool.xml > > 4.0K /var/lib/jenkins/hudson.plugins.sidebar_link.SidebarLinkPlugin.xml > > 4.0K /var/lib/jenkins/hudson.plugins.sonar.SonarGlobalConfiguration.xml > > 4.0K /var/lib/jenkins/hudson.plugins.timestamper.TimestamperConfig.xml > > 4.0K /var/lib/jenkins/hudson.tasks.Mailer.xml > > 4.0K /var/lib/jenkins/hudson.tasks.Shell.xml > > 4.0K /var/lib/jenkins/hudson.triggers.SCMTrigger.xml > > 4.0K /var/lib/jenkins/identity.key.enc > > 4.0K /var/lib/jenkins/io.jenkins.plugins.junit.storage.JunitTestResultStorageConfiguration.xml > > 4.0K /var/lib/jenkins/jenkins.fingerprints.GlobalFingerprintConfiguration.xml > > 4.0K /var/lib/jenkins/jenkins.install.InstallUtil.lastExecVersion > > 4.0K /var/lib/jenkins/jenkins.install.UpgradeWizard.state > > 4.0K /var/lib/jenkins/jenkins.model.ArtifactManagerConfiguration.xml > > 4.0K /var/lib/jenkins/jenkins.model.GlobalBuildDiscarderConfiguration.xml > > 4.0K /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml > > 4.0K /var/lib/jenkins/jenkins.plugins.msginject.MsgInjectConfig.xml > > 4.0K /var/lib/jenkins/jenkins.security.ResourceDomainConfiguration.xml > > 4.0K /var/lib/jenkins/jenkins.security.apitoken.ApiTokenPropertyConfiguration.xml > > 4.0K /var/lib/jenkins/jenkins.tasks.filters.EnvVarsFilterGlobalConfiguration.xml > > 4.0K /var/lib/jenkins/jenkins.telemetry.Correlator.xml > > 4.0K /var/lib/jenkins/net.plavcak.jenkins.plugins.scmskip.SCMSkipBuildWrapper.xml > > 4.0K /var/lib/jenkins/net.uaznia.lukanus.hudson.plugins.gitparameter.GitParameterDefinition.xml > > 4.0K /var/lib/jenkins/nodeMonitors.xml > > 4.0K /var/lib/jenkins/org.codefirst.SimpleThemeDecorator.xml > > 4.0K /var/lib/jenkins/org.jenkinsCi.plugins.projectDescriptionSetter.DescriptionSetterWrapper.xml > > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.ansible_tower.AnsibleTower.xml > > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.corsfilter.AccessControlsFilter.xml > > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.lucene.search.config.SearchBackendConfiguration.xml > > 4.0K /var/lib/jenkins/org.jenkinsci.plugins.workflow.flow.GlobalDefaultFlowDurabilityLevel.xml > > 4.0K /var/lib/jenkins/org.quality.gates.jenkins.plugin.GlobalConfig.xml > > 4.0K /var/lib/jenkins/queue.xml > > 4.0K /var/lib/jenkins/queue.xml.bak > > 4.0K /var/lib/jenkins/scriptApproval.xml > > 4.0K /var/lib/jenkins/secret.key > > 4.0K /var/lib/jenkins/sidebar-link.xml > > 12K /var/lib/jenkins/gerrit-trigger.xml > > 16K /var/lib/jenkins/hudson.plugins.ansicolor.AnsiColorBuildWrapper.xml > > 20K /var/lib/jenkins/config.xml > > 44K /var/lib/jenkins/nodes > > 72K /var/lib/jenkins/secrets > > 356K /var/lib/jenkins/users > > 1.7M /var/lib/jenkins/logs > > 1.8M /var/lib/jenkins/luceneIndex > > 3.6M /var/lib/jenkins/updates > > 3.9M /var/lib/jenkins/plugins_backup > > 316M /var/lib/jenkins/plugins > > 316M /var/lib/jenkins/plugins_not_working > > 372M /var/lib/jenkins/plugins_old_2ndFeb2024 > > 376M /var/lib/jenkins/plugins_working_2ndFeb2024 > > 1.8G /var/lib/jenkins/userContent > > 11G /var/lib/jenkins/jobs > > 19G /var/lib/jenkins/workspace > > > > # du -sh /fuego-rw > > 13G /fuego-rw > > > > # du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc list-runs | wc -l > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Dhrystone-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-aarch64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.GLMark-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.IOzone-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Interbench-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Java-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.OpenSSL-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Stream-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.Whetstone-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.bonnie-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.cyclictest-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.dbench4-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ebizzy-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.ffsb-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.fio-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.gtkperf-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.hackbench-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.himeno-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.iperf-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.linpack-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.lmbench2-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.nbench_byte-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netperf-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.netpipe-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.signaltest-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.tiobench-x86_64 > > 0 /fuego-rw/buildzone/ARM64-SSH-Test.default.Benchmark.x11perf-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Dhrystone-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.GLMark-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.IOzone-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Interbench-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Java-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.OpenSSL-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Stream-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.Whetstone-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.bonnie-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.cyclictest-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.dbench4-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ebizzy-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.ffsb-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.fio-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.gtkperf-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.hackbench-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.himeno-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.iperf-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.linpack-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.lmbench2-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.nbench_byte-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netperf-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.netpipe-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.signaltest-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.tiobench-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Benchmark.x11perf-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.LTP-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.OpenSSL-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.aiostress-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bc-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.bzip2-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.crashme-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.expat-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.fontconfig-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ft2demos-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.glib-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.hello_world-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.ipv6connect-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.jpeg-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.linus_stress-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.netperf-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.pi_tests-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.rmaptest-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.scrashme-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.stress-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.synctest-x86_64 > > 0 /fuego-rw/buildzone/X86-SSH-Test.default.Functional.zlib-x86_64 > > 4.0K /fuego-rw/buildzone/README > > 12K /fuego-rw/buildzone/Functional.bc-x86_64 > > 32K /fuego-rw/buildzone/Functional.hello_world-x86_64 > > 32K /fuego-rw/buildzone/Functional.linus_stress-x86_64 > > 32K /fuego-rw/buildzone/Functional.synctest-x86_64 > > 36K /fuego-rw/buildzone/Functional.rmaptest-x86_64 > > 48K /fuego-rw/buildzone/Benchmark.himeno-x86_64 > > 48K /fuego-rw/buildzone/Benchmark.linpack-x86_64 > > 48K /fuego-rw/buildzone/Functional.ipv6connect-x86_64 > > 60K /fuego-rw/buildzone/Benchmark.Stream-x86_64 > > 76K /fuego-rw/buildzone/Benchmark.Whetstone-x86_64 > > 76K /fuego-rw/buildzone/Benchmark.ebizzy-x86_64 > > 84K /fuego-rw/buildzone/Functional.aiostress-x86_64 > > 168K /fuego-rw/buildzone/Benchmark.Dhrystone-x86_64 > > 240K /fuego-rw/buildzone/Benchmark.tiobench-x86_64 > > 244K /fuego-rw/buildzone/Functional.crashme-x86_64 > > 488K /fuego-rw/buildzone/Benchmark.Interbench-x86_64 > > 576K /fuego-rw/buildzone/Functional.scrashme-x86_64 > > 760K /fuego-rw/buildzone/Benchmark.hackbench-x86_64 > > 816K /fuego-rw/buildzone/Benchmark.signaltest-x86_64 > > 868K /fuego-rw/buildzone/Functional.pi_tests-x86_64 > > 900K /fuego-rw/buildzone/Benchmark.netpipe-x86_64 > > 904K /fuego-rw/buildzone/Benchmark.cyclictest-x86_64 > > 1004K /fuego-rw/buildzone/Functional.stress-x86_64 > > 1.3M /fuego-rw/buildzone/Benchmark.ffsb-x86_64 > > 1.6M /fuego-rw/buildzone/Benchmark.GLMark-aarch64 > > 1.6M /fuego-rw/buildzone/Benchmark.GLMark-x86_64 > > 1.7M /fuego-rw/buildzone/Benchmark.nbench_byte-x86_64 > > 2.1M /fuego-rw/buildzone/Benchmark.iperf-x86_64 > > 2.3M /fuego-rw/buildzone/Benchmark.gtkperf-x86_64 > > 2.4M /fuego-rw/buildzone/Benchmark.bonnie-x86_64 > > 2.5M /fuego-rw/buildzone/Benchmark.x11perf-x86_64 > > 2.9M /fuego-rw/buildzone/Functional.bzip2-x86_64 > > 3.0M /fuego-rw/buildzone/Functional.zlib-x86_64 > > 3.1M /fuego-rw/buildzone/Benchmark.IOzone-x86_64 > > 3.5M /fuego-rw/buildzone/Functional.jpeg-x86_64 > > 5.2M /fuego-rw/buildzone/Benchmark.lmbench2-x86_64 > > 6.1M /fuego-rw/buildzone/Benchmark.netperf-x86_64 > > 6.1M /fuego-rw/buildzone/Functional.netperf-x86_64 > > 7.0M /fuego-rw/buildzone/Functional.fontconfig-x86_64 > > 9.1M /fuego-rw/buildzone/Benchmark.fio-x86_64 > > 29M /fuego-rw/buildzone/Functional.expat-x86_64 > > 33M /fuego-rw/buildzone/Benchmark.dbench4-x86_64 > > 34M /fuego-rw/buildzone/Functional.ft2demos-x86_64 > > 49M /fuego-rw/buildzone/Functional.glib-x86_64 > > 74M /fuego-rw/buildzone/Benchmark.OpenSSL-x86_64 > > 74M /fuego-rw/buildzone/Functional.OpenSSL-x86_64 > > 160M /fuego-rw/buildzone/Benchmark.Java-x86_64 > > 1.8G /fuego-rw/buildzone/X86-SSH-Test.default.Functional.kernel_build-x86_64 > > 2.4G /fuego-rw/buildzone/Functional.LTP-x86_64 > > > > Outside the container, can you run: > > $ docker image ls > > nxo-container-backup-2ndsep2021 latest b6a3d16508e8 2 years ago 173GB > > fuego-base-image-20thaug2019-container latest b2cf22891ac6 4 years ago 2.41GB > > fuegocontainerbackup-4thmar2019 latest 04112c383476 5 years ago 3.34GB > > fuego-piper-image-22ndnov2021 latest 0121b1c342e6 6 years ago 10.7GB > > fuego-xyz-image-22ndnov2021 latest 0bf6b942db70 6 years ago 5.8GB > > fuego-dart-image-22ndnov2021 latest 1d2c6a99dc0a 6 years ago 1.66GB > > fuego-pavv-image-22ndnov2021 latest ba71ede1e872 6 years ago 4.63GB > > fuego-gpu-image-22ndnov2021 latest 42718d86dd09 6 years ago 3.41GB > > > > Regarding below request, I just want to lock the Jenkins user > > interface/ dashboard, so that none can see all the jobs, results, logs and other details without logging in. So to start with even one user with > login credentials will help. > > But I don't know how Jenkins credentials relate to user accounts. I > > haven't ever set up Jenkins to handle multiple user accounts. Indeed, > > in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way > to create or manage new user accounts. > > > > Thanks a lot for all other details, they were very helpful. > > > > Regards, > > Dhinakar K. > > > > -----Original Message----- > > From: Bird, Tim [mailto:Tim.Bird@sony.com] > > Sent: Wednesday, March 13, 2024 5:47 AM > > To: dhinakar.k <dhinakar.k@samsung.com>; fuego+owner@lists.linux.dev; > > fuego@lists.linuxfoundation.org > > Subject: RE: HTML message rejected: Re: Distribute continuous > > integration (build) to servers outside docker container > > > > Hey Dhinakar, > > > > Sorry to not respond sooner. There was a lot to process in your > > email, and some of it I don't have answers for, but I'll try to give some status and feedback where I can. > > > > > -----Original Message----- > > > From: dhinakar.k <dhinakar.k@samsung.com> Hi Tim, > > > > > > I had sent a reply to your response on 3rd Feb.2024, not sure if it had reached you. > > > I am posting the same message below again. > > > > > > Thanks a lot for your prompt reply. > > > Iam looking forward to try the updated docker container with an > > > upgraded distribution base from Debian stretch to Debian bullseye, > > > and from a Jenkins version of 2.249.3 to 2.414.1. Please let me > > whenever it is available. > > > > OK - Will do. I haven't been able to allocate time for testing and > > fixing it, so I didn't want to push it to the master branch. If I pushed something to a testing branch, would you be able to download from > there and test it? > > > > One issue that came up is that some of the toolchains for some of my > > old boards/distros are not supported (ie don't work) in Debian bullseye, and so I was looking for solutions for that. > > > > > We are migrating to another server, so Iam looking for best options > > > to transfer all the content to the new server. Since the docker > > > container size is huge I want to trim it down by removing unwanted old logs. > > > > It would be good to know where the container size is coming from. Is > > it in the Fuego build directories, the Fuego run directories (where > > Fuego logs and run artifacts are stored), or in the Jenkins build > > directories (where Jenkins logs and artifacts are)? The materials under /fuego-rw should be on the host system, and not inside the container > itself (since these are volume mounts). But I don't know how this space is accounted to the container (if any). It shouldn't be in any container > image. > > > > The materials under /var/lib/jenkins are inside the container image. > > > > Can you run the following commands, inside your container, and send the results: > > > > # du -sh /var/lib/jenkins > > # du -sh /var/lib/jenkins/* | sort -h > > # du -sh /fuego-rw > > # du -sh /fuego-rw/buildzone/* | sort -h # du -sh /fuego-rw/logs/* | > > sort -h # ftc list-nodes | wc -l # ftc list-jobs | wc -l # ftc > > list-runs | wc -l > > > > Outside the container, can you run: > > $ docker image ls > > (and remove any lines not related to fuego images) > > > > But I don't know how Jenkins credentials relate to user accounts. I > > haven't ever set up Jenkins to handle multiple user accounts. Indeed, > > in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way > to create or manage new user accounts. > > > > > Also, I am worried about server crash and hard disk crash due to so many read/write cycles (that happens during continuous integration). > > > Hence looking for the best way to back up the container and also to > > > distribute the load & read/write cycles to another server (via an > > > ssh agent). I tried with plain Jenkins via ssh credentials plugin, > > > it seems feasible. Hence if fuego upgrades to debian bullseye it > > > would be really > > helpful. > > OK - I'll keep plugging away at this. > > > > > > > > Also, would like to know, when login credentials support will be > > > available in fuego? Because as a security measure we are supposed to > > > have login credentials for each user. Will be available after debian > > upgrade? > > > > I believe you are referring to Jenkins credentials? It appears that > > Jenkins supports credentials in version 2.414, which is the new version of Jenkins in the new docker container (with the Debian upgrade). So > I believe the answer is yes. > > > > But I don't know how Jenkins credentials relate to user accounts. I > > haven't ever set up Jenkins to handle multiple user accounts. Indeed, > > in my current 2.414.1 Jenkins setup I can see an option to list "People" (which I presume is registered user accounts), but I can find no way > to create or manage new user accounts. > > > > > With respect to removing the logs I will figure out the best way as > > > I see so many options but not sure if that will reduce the container > > > size directly or need to run some container commands to trim it > > > down. I > > will explore that as well and email my observations in this forum. > > > > I have considered adding an "rm-run" sub-command to ftc, which would allow you to remove old runs. > > However, I'd like to see if it's the run data that is causing the size problem that you are seeing. > > > > If you want to archive the data for an individual run, then you can > > use 'ftc package-run' and move the resulting package to some long-term > > storage device (before, ostensibly, removing that run from the > > system.) > > > > I hope this is helpful. > > -- Tim > > > > > > > > > > > > Regards, > > > Dhinakar K. > > > > > > -----Original Message----- > > > From: Bird, Tim [mailto:Tim.Bird@sony.com] > > > Sent: Saturday, February 3, 2024 3:44 AM > > > To: dhinakar k <dhinakar.k@gmail.com>; fuego+owner@lists.linux.dev; > > > fuego@lists.linuxfoundation.org > > > Subject: RE: HTML message rejected: Re: Distribute continuous > > > integration (build) to servers outside docker container > > > > > > Dhinakar, See my answers inline below.. > > > > > > > -----Original Message----- > > > > From: dhinakar k <dhinakar.k@gmail.com> Dear Fuego users, > > > > > > > > I have a requirement, so I thought of checking with the community > > > > if it was feasible and if so what is the best or most efficient > > > > way to implement it. > > > > We have fuego setup in a server and all our continuous integration > > > > jobs (for multiple code bases) are setup in it. But the issue is > > > > there are too many builds happening, which puts pressure on the > > > > hard disk (so many read/writes) and also increases the docker > > > > container size (because of build logs). > > > > > > > > Hence I would like to know if we can distribute the load to > > > > another server, for e.g. job will be created on our main server > > > > but build will happen on the agent on another server. The idea is > > > > to move out all cpu, memory, storage intensive activity to agents > > > > (other > > > > servers) on the network. The resultant logs can also be stored on > > > > those servers (outside docker container) so that docker size won't > > > > be growing rapidly. > > > > > > > > I tried creating a new node and setup an agent (connect via ssh) > > > > but facing issues in installing the 'SSH Build Agents' plugin > > > > because of compatibility issues (jenkins version needs to be upgraded). > > > > > > > > If anyone already tried and has a solution for this issue please let me know. > > > > > > Unfortunately, I have not tried this, so I'm not familiar with how Jenkins supports distributed jobs. > > > > > > But I have a few ideas to share. > > > > > > First, I'm working now on an update to the docker container, from a > > > distribution base of Debian stretch to Debian bullseye, and from a > > > Jenkins version of 2.249.3 to 2.414.1 (thanks to a patch provided by > > > Fuego user Yoichi Tachibana). I have applied the patches, but found > > some issues with the build instructions for some packages working in > > Debian bullseye. So I have not committed the changes to > > > the Fuego master branch yet. I hope to be able to do this soon. > > > > > > This might help with the issue of not being able to use the ssh agent plugin for Jenkins. > > > > > > Second, it may be possible to do some kind of proxy job submission > > > thing, by creating proxy jobs on the main Jenkins server, but having > > > the jobs actually executed ("built", in Jenkins terminology) on > > > secondary > > servers. > > > > > > There are a number of ways of supporting remote execution, and how > > > you implement it will determine where the CPU cycles and memory > > > usage is during the job, and where the logs end up living (and > > > taking up > > > space.) > > > > > > If you are using 'ttc' as your transport layer, recent versions of > > > ttc have support for remote boards via ssh (configured in ttc.conf). But that would leave logs on the master server. > > > > > > There are multiple ways to configure Fuego on the secondary servers > > > - for example using Docker containers or installing Fuego natively. > > > In any event, it may be possible to set up proxy jobs on the main > > > server that > > call 'ftc' on the secondary > > > servers, to run the jobs of interest. Depending on how this was structured > > > you may end up with build artifacts and logs on the secondary > > > system, and only Jenkins artifacts and logs on the main system (which should save space). > > > > > > It might be good to get a more detailed description of your Jenkins setup to be able to understand the options available. > > > > > > > > > > > Also, what is the best way to backup the container after trimming > > > > it down (remove unwanted files/logs etc. and reduce it's size)? > > > > > > I'm not sure how to backup a container in Docker. There's the 'docker container commit' > > > command, but I haven't used that. > > > > > > In general, to trim down you data usage you will want to perform a few operations. > > > Inside Jenkins, you can delete individual builds. You can likely > > > automate this (not have to do it manually through the UI), by using > > > the remote API. That should work for the material that is inside > > > the container. Lots of data, however, including the build artifacts > > > and the > > logs are natively on the host, and are only present in the container through bind mounts. > > > > > > This means you can remove this data directly using file operations (e.g. rm) on the host. > > > There are a few nuances to this, however, as some of the build directories are referenced via symlinks. > > > > > > 'ftc' currently supports removing jobs and nodes. I don't recall off the top of my head if this includes removing all the builds for a job or > not. > > > But in any case, it sounds like you only want to remove some of the builds/runs, not the jobs themselves. > > > We could possibly support 'ftc rm-run', which could remove specific runs. 'ftc query-runs' > > > could be used to generate a list of runs (e.g. that were created before a certain date). > > > > > > Please provide a few more details, and maybe we can talk through and/or develop some solutions that will help meet your needs. > > > > > > -- Tim > > > > > > > > > > > On Thu, Feb 1, 2024 at 11:52 PM <fuego+owner@lists.linux.dev> wrote: > > > > > > > > > > Greetings! > > > > > > > > > > This is the mlmmj program managing the <fuego@lists.linux.dev> > > > > > mailing list. > > > > > > > > > > Your message to <fuego@lists.linux.dev> was not delivered to the > > > > > list because it contained a HTML part. Only text/plain messages > > > > > are allowed on this list. > > > > > > > > > > Please configure your mail client to only send plain text mail. > > > > > > > > > > For your reference, the rejected message follows below. > > > > > > > > > > > > > > > > > > > > ---------- Forwarded message ---------- > > > > > From: dhinakar k <dhinakar.k@gmail.com> > > > > > To: fuego+help@lists.linux.dev, fuego@lists.linuxfoundation.org, > > > > > "Bird, Timothy" <Tim.Bird@sony.com> > > > > > Cc: > > > > > Bcc: > > > > > Date: Thu, 1 Feb 2024 23:51:30 +0530 > > > > > Subject: Re: Distribute continuous integration (build) to > > > > > servers outside docker container Dear Fuego users, > > > > > > > > > > I have a requirement, so I thought of checking with the > > > > > community if it was feasible and if so what is the best or most > > > > > efficient way to > > > > implement it. > > > > > We have fuego setup in a server and all our continuous > > > > > integration jobs (for multiple code bases) are setup in it. But > > > > > the issue is there are > > > > too many builds happening, which puts pressure on the hard disk > > > > (so many read/writes) and also increases the docker container size (because of build logs). > > > > > > > > > > Hence I would like to know if we can distribute the load to > > > > > another server, for e.g. job will be created on our main server > > > > > but build will > > > > happen on the agent on another server. The idea is to move out all > > > > cpu, memory, storage intensive activity to agents (other servers) > > > > on the network. The resultant logs can also be stored on those > > > > servers > > > (outside docker container) so that docker size won't be growing rapidly. > > > > > > > > > > I tried creating a new node and setup an agent (connect via ssh) > > > > > but facing issues in installing the 'SSH Build Agents' plugin > > > > > because of > > > > compatibility issues (jenkins version needs to be upgraded). > > > > > > > > > > If anyone already tried and has a solution for this issue please let me know. > > > > > > > > > > Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)? > > > > > > > > > > Regards, > > > > > Dhinakar K. > > > > > Samsung India, Bengaluru. > > > > > > > > > > On Thu, Feb 1, 2024 at 10:30 PM dhinakar k <dhinakar.k@gmail.com> wrote: > > > > >> > > > > >> Dear Fuego users, > > > > >> > > > > >> I have a requirement, so I thought of checking with the > > > > >> community if it was feasible and if so what is the best or most > > > > >> efficient way to > > > > implement it. > > > > >> We have fuego setup in a server and all our continuous > > > > >> integration jobs (for multiple code bases) are setup in it. But > > > > >> the issue is there are > > > > too many builds happening, which puts pressure on the hard disk > > > > (so many read/writes) and also increases the docker container size (because of build logs). > > > > >> > > > > >> Hence I would like to know if we can distribute the load to > > > > >> another server, for e.g. job will be created on our main server > > > > >> but build will > > > > happen on the agent on another server. The idea is to move out all > > > > cpu, memory, storage intensive activity to agents (other servers) > > > > on the network. The resultant logs can also be stored on those > > > > servers > > > (outside docker container) so that docker size won't be growing rapidly. > > > > >> > > > > >> I tried creating a new node and setup an agent (connect via > > > > >> ssh) but facing issues in installing the 'SSH Build Agents' > > > > >> plugin because of > > > > compatibility issues (jenkins version needs to be upgraded). > > > > >> > > > > >> If anyone already tried and has a solution for this issue please let me know. > > > > >> > > > > >> Also, what is the best way to backup the container after trimming it down (remove unwanted files/logs etc. and reduce it's size)? > > > > >> > > > > >> Regards, > > > > >> Dhinakar K. > > > > >> Samsung India, Bengaluru. > > > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-03-22 22:28 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CGME20240315153759epcas5p3d49d4d0bfcf86c98f2c3754afa96034b@epcas5p3.samsung.com>
2024-03-15 15:37 ` RE[3]: Distribute continuous integration (build) to servers outside docker container dhinakar.k
2024-03-15 18:41 ` Bird, Tim
2024-03-17 14:27 ` dhinakar.k
2024-03-22 22:28 ` Bird, Tim
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.