Jenkins – Troubleshooting Limited Error Output in Failing Builds

gitJenkins

I have recently setup a gogs server to connect with Jenkins (previously we were using SVN). The connection is working but the build fails when retrieving objects from the gogs server around 41%-43% (different for each build but always within that range).

Is there some kind of limit set by the GOGS api? Below is the only error output I am getting.

Receiving objects:  41% (270082/645355), 2.68 GiB | 18.83 MiB/s   
Receiving objects:  41% (270419/645355), 2.70 GiB | 18.96 MiB/s   
Killed by signal 15.

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1715)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405)
    at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:351)
    at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:196)
    at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:172)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:107)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:271)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

UPDATE:
The server my jenkins master is on needed a disk size increase (the android ROM build I am doing is absolutely massive and as filling up the cache to the point of failure). I have added another 60GB of space to my setup and re-ran my build.

As Jey mentioned in the comments, ssh may have a role to play with this so I added the following lines to sshd_config on the Jenkins master to satisfy the need for a long open connection:

ClientAliveInterval 120
ClientAliveCountMax 720

The build did make it a bit further than previous attempts but still failed in the end, in the same manner as before:

Receiving objects:  55% (355794/645358), 3.51 GiB | 13.63 MiB/s   
Receiving objects:  55% (359617/645358), 3.52 GiB | 13.75 MiB/s   
Killed by signal 15.

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1715)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405)
    at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:351)
    at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:196)
    at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:172)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:107)
    at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:271)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

New update:

Doing some more testing (don't believe it's a network issue but thanks for the comment). I noticed the build is being killed after 10 minutes exactly. From reviewing others with the same problem I found that you can increase this through advanced options. I have done this but it's still failing after 10 minutes exactly.

enter image description here

NEW UPDATE:

I have attached a copy of my build config.xml, as you can see the timeout has been updated in the file but the build is still using the default timeout limit of 10 minutes. Not sure what I'm missing here…

<?xml version='1.1' encoding='UTF-8'?>
<flow-definition plugin="workflow-job@2.17">
  <actions/>
  <description>Android ROM Build</description>
  <keepDependencies>false</keepDependencies>
  <properties>
    <hudson.plugins.buildblocker.BuildBlockerProperty plugin="build-blocker-plugin@1.7.3">
      <useBuildBlocker>false</useBuildBlocker>
      <blockLevel>GLOBAL</blockLevel>
      <scanQueueFor>DISABLED</scanQueueFor>
      <blockingJobs></blockingJobs>
    </hudson.plugins.buildblocker.BuildBlockerProperty>
    <org.jenkinsci.plugins.gogs.GogsProjectProperty plugin="gogs-webhook@1.0.14">
      <gogsSecret>XXXXXXXXXXXXX</gogsSecret>
      <gogsUsePayload>false</gogsUsePayload>
    </org.jenkinsci.plugins.gogs.GogsProjectProperty>
  </properties>
  <definition class="org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition" plugin="workflow-cps@2.47">
    <scm class="hudson.plugins.git.GitSCM" plugin="git@3.9.1">
      <configVersion>2</configVersion>
      <userRemoteConfigs>
        <hudson.plugins.git.UserRemoteConfig>
          <url>gogs@xxx-xxx.xxx.com:git-admin/SF-Firefly-RK3399.git</url>
          <credentialsId>XXXXXXXXXXXXXX</credentialsId>
        </hudson.plugins.git.UserRemoteConfig>
      </userRemoteConfigs>
      <branches>
        <hudson.plugins.git.BranchSpec>
          <name>*/fresh_after_opersys_patches</name>
        </hudson.plugins.git.BranchSpec>
      </branches>
      <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
      <submoduleCfg class="list"/>
      <extensions>
        <hudson.plugins.git.extensions.impl.CloneOption>
          <shallow>true</shallow>
          <noTags>false</noTags>
          <reference>gogs@xxx-xxx.xxx.com:git-admin/SF-Firefly-RK3399.git</reference>
          <timeout>240</timeout>
          <depth>1</depth>
          <honorRefspec>true</honorRefspec>
        </hudson.plugins.git.extensions.impl.CloneOption>
        <hudson.plugins.git.extensions.impl.CheckoutOption>
          <timeout>240</timeout>
        </hudson.plugins.git.extensions.impl.CheckoutOption>
      </extensions>
    </scm>
    <scriptPath>Jenkinsfile</scriptPath>
    <lightweight>true</lightweight>
  </definition>
  <triggers/>
  <disabled>false</disabled>
</flow-definition>

Best Answer

Ok, so after banging my head for hours on my desk I finally figured it out. The issue was the lightweight checkout flag on the pipeline build. Once it was disabled (as shown in the screen shot below, the 10 minute timeout was increased to what I had set in the advanced clone/fetch behaviour.

Lightweight checkout flag

Related Topic