Bug - Fixed Jenkins freezing during tests only sometimes

[fewyn]

This is more of a tracking thing for myself but Jenkins has been freezing for awhile only sometimes during the testing of builds. Happened this morning with 26080, I have had a bandaid in place for awhile now where if Jenkins becomes unresponsive after 15 minutes it'll automatically restart the Jenkins service.

So right now I've added -d to the testing for gradle, hopefully this will give me some more insight on why it's freezing.

 

[fewyn]

And of course since adding it nothing has frozen except now it's doing it on builds... added a debug flag to the jar build portion to hopefully help identifying the culprit.

 

[fewyn]

Not sure if anyone else might be more familiar with Gradle but we had it freeze today and here's the log: https://gist.github.com/fewyn/749b6a09209a4290ae833a378ab1da6c

I thought it might be a memory issue but what I've seen that's just normal health checks. Any ideas out there?

 

[fronobulax]

Can't help. The final error is associated with a compile time warning which we see every time we do a fresh compile and are unlikely to fix because it is in 3rd party source that we really want to abandon, not maintain. Perhaps something in your environment is treating that warning as a fatal error? Perhaps something is unspecified in Gradle so Jenkins is providing?

 

[heeheehee]

it reads to me like your system is ooming

2022-01-06T17:22:26.989+0000 [DEBUG] [org.gradle.process.internal.health.memory.DefaultMemoryManager] 411758592 memory requested, 0 released, 89534464 free
2022-01-06T17:22:31.990+0000 [DEBUG] [org.gradle.process.internal.health.memory.DefaultMemoryManager] 411758592 memory requested, 44896256 free
2022-01-06T17:22:31.992+0000 [DEBUG] [org.gradle.workers.internal.WorkerDaemonExpiration] Will attempt to release 392 of memory
2022-01-06T17:22:31.992+0000 [DEBUG] [org.gradle.process.internal.health.memory.DefaultMemoryManager] 411758592 memory requested, 0 released, 44896256 free
...
2022-01-06T17:25:01.704+0000 [DEBUG] [org.gradle.process.internal.health.memory.DefaultMemoryManager] 411758592 memory requested, 0 free
2022-01-06T17:25:01.704+0000 [DEBUG] [org.gradle.workers.internal.WorkerDaemonExpiration] Will attempt to release 392 of memory
2022-01-06T17:25:01.707+0000 [DEBUG] [org.gradle.process.internal.health.memory.DefaultMemoryManager] 411758592 memory requested, 0 released, 0 free

 

[fewyn]

I made some configuration changes and added a swap file to the server, we'll see if this helps alleviate issues.

 

[fewyn]

Increasing the heap size for the java process and adding a swap file seems to have resolved this. I've removed debug mode from the Jenkins build process.