vFunction Server cannot complete an upgrade after Disk Space issues
Overview of the Issue
This issue occurs in the following circumstances:
- A vFunction Server becomes unresponsive
- An investigation of the VM where the Server runs shows that the VM is out of Disk Space and that some of the vFunction Docker Containers have crashed
- After freeing up Disk Space, the upgrade.sh script is run to restore the vFunction Containers
- The upgrade.sh script gets stuck and displays the following console output:
2024-06-25 13:34:47.953 Start upgrading server to version v3.4.1302
2024-06-25 13:34:48.768 Creating dedicated network "vfunction"...
2024-06-25 13:34:48.864 Using network vfunction with gateway 172.18.0.1
2024-06-25 13:34:48.961 Stopping all dockers...
2024-06-25 13:34:48.993 vfunction-monit...
2024-06-25 13:34:49.144 vfunction-vfapi-users...
2024-06-25 13:34:49.265 vfunction-vfapi-organizations...
- Listing all Docker Containers shows that one or more are unhealthy
CONTAINER ID IMAGE NAMES COMMAND CREATED STATUS PORTS
bb17b4582f1a vfunction/vfunction-vfapi-organizations-alpine:v3.4.1302 vfunction-vfapi-organizations /bin/bash /opt/vfun… 5 weeks ago Up 5 weeks (unhealthy) 0.0.0.0:8001->8001/tcp, :::8001->8001/tcp
The following steps can be used to resolve this issue:
- Stop and Start Docker and the Docker Sockets
systemctl stop docker
systemctl stop docker.socket
systemctl start docker
- Stop the Monit Container and any Unhealthy Containers. Remove the Unhealthy Container
docker stop vfunction-monit
docker stop vfunction-vfapi-organizations
docker rm -f vfunction-vfapi-organizations
- Run the upgrade.sh script again to successfully restart all Containers
bash /opt/vfunction/server-installation/upgrade.sh