Podman Container's ctr.log uses all disk space

Overview of the Issue

This issue occurs in the following circumstances:

  1. The vFunction organization deploys the vFunction Server using Podman
  2. Unexpectedly, the vFunction Server UI becomes unresponsive
  3. In an SSH session, the disk space where the vFunction Server is installed appears to be out of disk space
  4. Further analysis through du -shx * shows that one Container’s ctr.log is using an unexpectedly high amount of disk space:

NOTE: Podman uses the ctr.log as the Container’s Console Output of stdout and stderr. This is all of the output from “podman logs $CONTAINER_ID”

ls -l /vfunction/storage/overlay-containers/$UUID/userdata
-rw-------.  $user $group  31916769280  Sep 30 04:09 ctr.log

Steps to Resolve the Issue

This issue can be resolved by clearing out the log file, setting the size limit to a smaller value, and running the vFunction Server upgrade:

  1. SSH to the vFunction Server VM
  2. Navigate to the directory with the large ctr.log
  3. Overwrite the existing log file:
echo "" > ctr.log
  1. Modify the Max Log Size value:
vi $HOME/.config/containers/containers.conf

### Add the lines below to this file to set the max size to 25mb
[containers]
max_log_size=26214400
  1. Run the vFunction Server Upgrade:
bash $BASE_DIR/vfunction/opt/vfunction/server-installation/upgrade.sh

Updated on 01 Oct 2024