[Fixed]-Django on Kubernetes Deployment: Best practices for DB Migrations

9👍

Well you are right about the part that multiple migrate commands will run against your database by multiple pods getting started.

But this will not cause any problems. When you are going to make actual changes to your database, if the changes are already applied, your changes will be ignored. So, say 3 pods start at the same time and run the migrate command. Only One of those commands will end up applying changes to the database. Migrations normally need to lock the database for different actions (this is highly related to your DBMS). The lock will happen by one of the migrate commands (one of the pods) and other commands should wait until the work of the first one is over. After the job is done by the first one, others’ commands will be ignored automatically. So each migration will happen once.

You can however, change your deployment strategy and ask kubernetes to first, spin up only 1 pod and when the first pod’s health check succeeds, others will spin up too. In this case, you can be sure that the lock time for the migration, will happen only once and others will just check that migrations are already applied and ignore them automatically.

👤nima

1👍

You may use Kubernetes init containers, which are specialized containers that run before app containers in a Pod. The init containers stop after successfully executing the commands you want, so they won’t occupy unnecessary resources.
Here is the official link:
https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

1👍

As I understand In K8s you can’t control the order of the pods.
Let’s say your DB host is a docker container (Dev environment) and not an RDS/ static host, migrations command could not be executed before the DB is up and ready.

I try to to use the ready function in app.py to execute run_migrations.py that execute shell script of run migrations but run migrations trigger app.py and it’s infinity loop.

In my organization We use circleCI which build the image first and then push it to AWS ECR with tags before deploy it to EKS. At the build stage I can’t assume the DB pod is ready so run migrations will cause an error.

When the DB for Django is also a K8s pod, the best way to execute it will be a job.

0👍

The best thing you can do is to use kind:Jobs, an run this before the rollout.
Please check: Django migrations by Kubernetes Job and persistent Volume Claim

Leave a comment