Error '[Backup]: Couldn't clean up validation database restore_validate_idp' prevents MySQL Pod from starting during upgrade
Overview of the Issue
This issue occurs in the following circumstances:
- An organization runs the upgrade.sh script to upgrade the vFunction Server in a Kubernetes Cluster via Helm Charts
- The MySQL Pod gets into a CrashLoopBackoff with the following error in the logs:
2025-09-09 06:03:11+00:00 [Note] [Entrypoint]: Creating a dump of DB before upgrade
2025-09-09 06:03:32+00:00 [Info] [Backup]: Validating dump /opt/vfunction/backups/dump-20250909060311-v4.3.1784-pre-upgrade-to-v4.3.1799.sql.tgz
2025-09-09 06:04:03+00:00 [Info] [Backup]: Cleaning up old validation databases before validation started...
2025-09-09 06:04:03+00:00 [Error] [Backup]: Couldn't clean up validation database restore_validate_idp
2025-09-09 06:04:03+00:00 [Error] [Backup]: There was an error during dump validation!
2025-09-09 06:04:03+00:00 [Error] [Entrypoint]: Dump of DB before upgrade created, but failed to validate it.
+ [[ -v IGNORE_DEFECTIVE ]]
+ rm -rf /opt/vfunction/errors/error-startup-20250909060542-v4.3.1799.log
+ exit 1
- Adding the following Keys to the mysql-custom-env.list configMap allows the MySQL Pod to come up healthy as expected
SKIP_DUMP: 'true'
IGNORE_DEFECTIVE: 'true'
- Attempting to manually run the backup.sh script on the MySQL Pod throws the following error
ERROR 3681 (HY000) at line 1: Schema 'restore_validate_idp' does not exist, but schema directory './restore_validate_idp/' was found. This must be resolved manually (e.g. by moving the schema directory to another location).
2025-09-11 06:19:08+00:00 [Error] [Backup]: Couldn't clean up validation database restore_validate_idp
Steps to Resolve the Issue
Take the following steps to resolve this issue:
- Exec to the MySQL Pod
- Move the /opt/vfunction/db/restore_validate_idp folder to a backup location
- Run the backup.sh script again to confirm it completes as expected
- Delete the restore_validate_idp folder