Java Application hangs while running Batch Jobs
Overview of the Issue
This issue occurs in the following circumstances:
- An organization adds the vFunction Agent to Batch Job scripts that run as part of a Java Application
- When the Batch Job is run, the script never completes and never returns to the Prompt
- The Application logging shows no unexpected logging
- Adding debug flags for vFunction shows no unexpected logging in the vfagent.log, vfagent.log.beanstracking nor vfagent.log.dbtracking
Steps to Gather Additional Information
- Open the vFunction instance’s native-conf.json and turn off stack_sampling. For example
- Open the native-conf.json in a text editor. For example
vi /etc/sysconfig/vfunction/agent/instances/default-java/native-conf.json
- Remove the comment in front of stack_sampling: false. For example
After the change
{
"#disable_thread_following": true,
"stack_sampling": false,
"#sampling_ring_size": 256,
"#disable_agct": true,
"#tracking_conf": {
"sockets": true,
"files": true,
"sync_objects": true
},
"#static_root_classes": {
"inclusions": [
"Lcom/example/OneClassToInclude;",
"Lcom/example/package1/"
],
"exclusions": [
"Lcom/example/"
]
}
}
- Modify the vFunction Agent Java Options to turn off the HTTP Agent and set the Number of Parsing Threads to 0
After the change
VFUN_OPTS="-agentpath:/opt/vfunction/agent/libJVMTIAgent.so=conf=/etc/sysconfig/vfunction/agent/instances/default-java/native-conf.json -javaagent:/opt/vfunction/agent/runtime-agent.jar=ConfigLocation=/etc/sysconfig/vfunction/agent/instances/default-java/vfunction-runtime-override.properties -Dcom.vfunction.runtime.shared.folder=/opt/vfunction/controller/instances/default-java/temp/ -Dcom.vfunction.runtime.offload.tmpFolder=/opt/vfunction/controller/instances/default-java/temp/ -Dcom.vfunction.logging.logFile=/var/log/vfunction/instances/default-java/vfagent.log -Dcom.vfunction.agent.http.enabled=false -Dcom.vfunction.jdbc.tracking.SqlParsingService.numParsingThreads=0"
Before the change
VFUN_OPTS="-agentpath:/opt/vfunction/agent/libJVMTIAgent.so=conf=/etc/sysconfig/vfunction/agent/instances/default-java/native-conf.json -javaagent:/opt/vfunction/agent/runtime-agent.jar=ConfigLocation=/etc/sysconfig/vfunction/agent/instances/default-java/vfunction-runtime-override.properties -Dcom.vfunction.runtime.shared.folder=/opt/vfunction/controller/instances/default-java/temp/ -Dcom.vfunction.runtime.offload.tmpFolder=/opt/vfunction/controller/instances/default-java/temp/ -Dcom.vfunction.logging.logFile=/var/log/vfunction/instances/default-java/vfagent.log"
- Run the Batch Job
- If the Application hangs, create a thread dump in another SSH session using JStack from the same JDK Bin directory used in the Batch file
### Find the Process ID (PID) of the running Java Process
ps -ef | grep java
### Replace $PID_OF_JAVA_PROCESS with the Process ID found using the command above
jstack $PID_OF_JAVA_PROCESS